Cassandra ZStandrad Compressor

Facebook has released it's new open sourced compression algorithm which is quite efficient and fast at the same time. To read more about it please look here or here.

It's actually so efficient that it seems to be a great candidate for Cassandra table compression.

So to save you all the required work, here I've implemented an Cassandra Zstandard compressor that is ready to use: https://github.com/MatejTymes/cassandra-zstd

Missing matched documents on searches and updates reproduction

This blog recently exposed an interesting concurrency caveat related to MongoDB where matching documents won't be found (or updated) if they are being reindexes.

The only part in the entry I was missing is a way how to reproduce this issue. So I decided to create a test which you can test against your version of MongoDB to check if it is still a problem.

Here it is:

package co.uk.matejtymes.mongodb;

import com.mongodb.*;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.util.*;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;

import static com.mongodb.BasicDBObjectBuilder.start;
import static java.util.Arrays.asList;
import static java.util.UUID.randomUUID;
import static java.util.concurrent.Executors.newFixedThreadPool;
import static java.util.concurrent.TimeUnit.SECONDS;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.Matchers.hasSize;
import static org.junit.Assert.assertThat;

public class ReindexFailureTest {

    private static final String STATE_FIELD = "state";

    private static final Random RANDOM = new Random();

    private DBCollection coll;

    @Before
    public void setUp() throws Exception {
        // todo: provide connection details for your mongoDB instance
        MongoClient mongo = new MongoClient("localhost", 27017);
        DB db = mongo.getDB("testDb");

        coll = db.getCollection("indexTest");
    }

    @After
    public void tearDown() throws Exception {
        coll.drop();
    }

    @Test
    public void shouldFindAllMatchingItemsEvenWhenRecalculatingIndex()throws Exception {
        int docCount = 250;
        int concurrentUpdates = 40;
        int attemptsCount = 1_000;

        List<String> stateValues = asList("Active", "Inactive");

        coll.createIndex(new BasicDBObject(STATE_FIELD, 1));

        List<String> allIds = createNDocumentsWithState(docCount, stateValues);

        ExecutorService executor = newFixedThreadPool(concurrentUpdates);
        for (int attempt = 1; attempt <= attemptsCount; attempt++) {
            System.out.println(attempt + ". attempt");

            CountDownLatch beginLatch = new CountDownLatch(concurrentUpdates + 1);
            CountDownLatch endLatch = new CountDownLatch(concurrentUpdates + 1);

            for (int update = 0; update < concurrentUpdates; update++) {
                executor.submit(() -> updateState(pickRandomItem(allIds), stateValues, beginLatch, endLatch));
            }

            List<String> foundIds = findDocumentsInState(stateValues, beginLatch, endLatch);

            Set<String> uniqueIds = new HashSet<>();
            Set<String> duplicateIds = new HashSet<>();
            Set<String> missingIds = new HashSet<>(allIds);

            for (String foundId : foundIds) {
                if (uniqueIds.contains(foundId)) {
                    duplicateIds.add(foundId);
                }
                uniqueIds.add(foundId);
                missingIds.remove(foundId);
            }

            if (!missingIds.isEmpty()) {
                System.err.println(missingIds.size() + ". missingIds: " + missingIds);
            }
            if (!duplicateIds.isEmpty()) {
                System.err.println(duplicateIds.size() + ". duplicateIds: " + duplicateIds);
            }

            assertThat(foundIds, hasSize(allIds.size()));
            assertThat(missingIds, hasSize(0));
            assertThat(duplicateIds, hasSize(0));
        }

        executor.shutdown();
        executor.awaitTermination(3, SECONDS);
    }

    @Test
    public void shouldUpdateAllMatchingItemsEvenWhenRecalculatingIndex()throws Exception {
        int docCount = 250;
        int concurrentUpdates = 40;
        int attemptsCount = 1_000;

        List<String> stateValues = asList("Active", "Inactive");

        coll.createIndex(new BasicDBObject(STATE_FIELD, 1));

        List<String> allIds = createNDocumentsWithState(docCount, stateValues);

        ExecutorService executor = newFixedThreadPool(concurrentUpdates);
        for (int attempt = 1; attempt <= attemptsCount; attempt++) {
            System.out.println(attempt + ". attempt");

            String fieldToUpdate = "field" + attempt;
            Object valueToSet = true;

            CountDownLatch beginLatch = new CountDownLatch(concurrentUpdates + 1);
            CountDownLatch endLatch = new CountDownLatch(concurrentUpdates + 1);

            for (int update = 0; update < concurrentUpdates; update++) {
                executor.submit(() -> updateState(pickRandomItem(allIds), stateValues, beginLatch, endLatch));
            }

            BasicDBObject query = new BasicDBObject(STATE_FIELD, new BasicDBObject("$in", stateValues));
            BasicDBObject update = new BasicDBObject("$set", new BasicDBObject(fieldToUpdate, valueToSet));

            beginLatch.countDown();
            int n = coll.updateMulti(query, update).getN();
            endLatch.countDown();


            List<String> updatedIds = new ArrayList<>();
            coll.find(new BasicDBObject(fieldToUpdate, valueToSet)).forEach(
                    dbObject -> updatedIds.add((String) dbObject.get("_id"))
            );

            Set<String> missingIds = new HashSet<>(allIds);
            missingIds.removeAll(updatedIds);


            if (!missingIds.isEmpty()) {
                System.err.println(missingIds.size() + ". missingIds: " + missingIds);
            }
            if (n != allIds.size()) {
                System.err.println("n = " + n);
            }
            if (updatedIds.size() != allIds.size()) {
                System.err.println("updateIds = " + updatedIds.size());
            }

            assertThat(n, equalTo(allIds.size()));
            assertThat(updatedIds, hasSize(allIds.size()));
            assertThat(missingIds, hasSize(0));
        }

        executor.shutdown();
        executor.awaitTermination(3, SECONDS);
    }

    /* ====================== */
    /* --- helper methods --- */
    /* ====================== */

    private List<String> createNDocumentsWithState(int docCount, List<String> stateValues) {
        List<String> ids = new ArrayList<>();

        for (int i = 0; i < docCount; i++) {
            String id = randomUUID().toString();
            String state = stateValues.get(i % stateValues.size());

            DBObject dbObject = start()
                    .add("_id", id)
                    .add(STATE_FIELD, state)
                    .get();
            coll.insert(dbObject);

            ids.add(id);
        }
        return ids;
    }

    private void updateState(String id, List<String> stateValues, CountDownLatch beginLatch, CountDownLatch endLatch) {
        BasicDBObject query = new BasicDBObject("_id", id);

        String oldStateValue = (String) coll.find(query).next().get(STATE_FIELD);
        String newStateValue = stateValues.stream().filter(state -> !state.equals(oldStateValue)).findFirst().get();

        BasicDBObject update = new BasicDBObject("$set", new BasicDBObject(STATE_FIELD, newStateValue));

        beginLatch.countDown();
        coll.update(query, update);
        endLatch.countDown();
    }

    private List<String> findDocumentsInState(List<String> stateValues, CountDownLatch beginLatch, CountDownLatch endLatch) {
        BasicDBObject query = new BasicDBObject(STATE_FIELD, new BasicDBObject("$in", stateValues));
        Iterator<DBObject> dbObjects = coll.find(query).iterator();

        List<String> foundIds = new ArrayList<>();

        beginLatch.countDown();;
        while (dbObjects.hasNext()) {
            foundIds.add((String) dbObjects.next().get("_id"));
        }
        endLatch.countDown();

        return foundIds;
    }

    private static <T> T pickRandomItem(List<T> values) {
        return values.get(RANDOM.nextInt(values.size()));
    }
}

Executor that notifies you when task finish

Java Executors don't let you know when all tasks are finished or to be more precise, don't block you until the tasks are finished. You could call shutdown() on them and then awaitTermination(), but this way you can't reuse the executor anymore, which is not great. This is why I create a class Runner that can accomplish this. It's used like this:

Runner runner = Runner.runner(10);

runner.runIn(2, SECONDS, runnable);
runner.run(runnable);


runner.waitTillDone(); // blocks until all tasks are finished (or failed)


// and reuse it

runner.runIn(500, MILLISECONDS, callable);

runner.waitTillDone();

runner.shutdownAndAwaitTermination();

The code for it can be found here:

https://github.com/MatejTymes/JavaFixes

Hope this will help

End of BigDecimal BSting

BigDecimal is so close to being great until you face few things which make you just scream.

• values are sometimes not equal (although you would like them to be):

    // yes: -1.2 is not equal to -1.20
    assertThat(new BigDecimal("-1.2").equals(new BigDecimal("-1.20")), is(false));

maybe this seems harmless but once your test will start to fail because the actual and expected domain object (using BigDecimal) are not equal although they are, you will just ask your self: why do we have to go trough this?

Also have you ever been paring with somebody on an interview, where the candidate told you that we should use BigDecimal for this interest rate calculation, but in the end you both decided not to do it - as the interview is not long enough - really BigDecimal adds aditional 10 to 30 minutes to inteview excercise as you have to deal with the equals method - and THIS is the STANDARD!!!

• equals is bad, but hashCode is even worse

    // yes: hashCodes for -1.2 and -1.20 are not the same as well
    assertThat(new BigDecimal("-1.2").hashCode(), is(not(new BigDecimal("-1.20").hashCode())));

but why would that even matter? Well, if you ever decide to make BigDecimal key for a HashMap, than a situation might occur where you won't find any value for your number, as their hashCode won't match.

• the ways how to create BigDecimal are not unified at all (depending on your originating value there are few different ways how you create it)

    new BigDecimal("-1.20"); // from string

    BigDecimal.valueOf(-120L, 2); // from long

    new BigDecimal(BigInteger.valueOf(-120L), 2); // from BigInteger

    // you should not create BigDecimal from float or double as you might get really weird value (because of transition from binary to decimal form)

I assumed that this is going to be fixed as this problems were present for many years, but it seems this is the design we'll have to live with.

This is why I decided to create a rewrite of BigDecimal called Decimal which you can find on this page (currently I'm finalizing the implementation):

https://github.com/MatejTymes/JavaFixes

It provides few advantages over BigDecimal

• unified creation using two possible factory methods (one more readable decimal(...), one shorter d(...))
• fixed equals
• fixed hashCode:

    // equals now works
    assertThat(decimal("-1.2").equals(decimal("-1.200")), is(true));
    assertThat(d("-1.2").equals(d("-1.200")), is(true));

    // and surprisingly hashCode as well
    assertThat(d("-1.2").hashCode(), is(d("-1.20").hashCode()));

• also the creation approach is always the same

    decimal("-1.20"); // from string

    decimal(-120L, 2); // from long

    decimal(BigInteger.valueOf(-120L), 2); // from BigInteger

    // you are not able to create Decimal from float or double but have to transform them into string first - otherwise you might get surprising values

• you can use underscores in the numbers to make them more readable

    Decimal value = d("-125_550_00.00"); // using underscores as you can use in java numbers

The Decimal is an abstract class currently extended by two implementations: LongDecimal - if value can be backed by long and HugeDecimal - backed by BigInteger (for all other numbers). You can't address them directly, but the library handles the transition between these types seamlessly while you're calling arithmetic operations.

And that's it. Please let me know if you can think of any other improvements, or just what you feel about this. I would be happy to hear your thoughts.

WebDriver fix for UnreachableBrowserException

This page is a summary of 3 different fixes for 3 different ways how to get UnreachableBrowserException-s while using WebDrivers

1. Caused by: java.net.SocketException: Software caused connection abort: recv failed

To fix this issue just replace usage of PhantomJSDriver with following FixedPhantomJSDriver:

public class FixedPhantomJSDriver extends PhantomJSDriver {

    private final int retryCount = 2;

    public FixedPhantomJSDriver() {
    }

    public FixedPhantomJSDriver(Capabilities desiredCapabilities) {
        super(desiredCapabilities);
    }

    public FixedPhantomJSDriver(PhantomJSDriverService service, Capabilities desiredCapabilities) {
        super(service, desiredCapabilities);
    }

    @Override
    protected Response execute(String driverCommand, Map<String, ?> parameters) {
        int retryAttempt = 0;

        while (true) {
            try {

                return super.execute(driverCommand, parameters);

            } catch (UnreachableBrowserException e) {
                retryAttempt++;
                if (retryAttempt > retryCount) {
                    throw e;
                }
            }
        }
    }
}

So in summary:

org.openqa.selenium.remote.UnreachableBrowserException: Error communicating with the remote browser. It may have died.

caused by:

Caused by: java.net.SocketException: Software caused connection abort: recv failed

(the recv failed is important)
happens (from my investigations) when connection is closed prematurely. In our case it seemed to be caused by ssl certificate handling in java (I'm still investigating this) and is extremely random. Luckily all http traffic is handled by the execute method. So by overriding it and adding simple retry functionality you provide a working workaround solution (it helped us on our project as we never had a failing/flaky tests again).
Although this is a specific implementation for PhantomJS driver, the same approach should work for other drivers as well.

2. Caused by: java.net.SocketTimeoutException: Read timed out

But there is still chance that you have different symptoms and your browser just hangs for some time until it throws:

Caused by: java.net.SocketTimeoutException: Read timed out

If you're working on a Windows machine, the chances are that you've reached limit of possible open connections. This normally happens, because Selenium creates a lot of connections and Windows keeps them opened/cached even when java triggered a close connection command. To fix this issue you need to change Windows registry values under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

you need to set/create two DWORD values:

MaxUserPort = 32768
TcpTimedWaitDelay = 30

MaxUserPort will increase the limit of possible open connections (you can select any value between 5000-65534, the higher the better). TcpTimedWaitDelay makes sure that windows will close stale connections (already closed by java) after 30 seconds (can't be set to lower, but without setting this the default value is 4 minutes !!!). In most cases this should fix your "hang" issue.

3. When your test still hangs for 3 hours until it fails

Unfortunately there is still a small chance that you have issues where your test will get stuck for 3 hours !!! The reason for this is that the HttpClientFactory in selenium has hardcoded socket timeout to 3 hours, and although there is a proposed fix to the selenium core code, until it will be accepted there is no way how to change it using normal means. For those who unfortunatelly must bear the pain, here is how you use my hacky, but working workaround to this problem:

public class FixExample {

    public static void main(String[] args) {
    
        // this is my custom workaround

        HttpParamsSetter.setSoTimeout(60 * 1000); // set socket timeout to 1 minute
        
        // and here goes your custom code

        WebDriver driver = new PhantomJSDriver();
        
        ...

        driver.quit();
    }
}

Here you can find the code that does the magic (this works because fields HttpCommandExecutor.httpClientFactory and HttpClientFactory.client are static fields that are initialized only once):

import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.params.HttpParams;
import org.openqa.selenium.remote.HttpCommandExecutor;
import org.openqa.selenium.remote.internal.HttpClientFactory;
import java.lang.reflect.Field;

public class HttpParamsSetter {

    @SuppressWarnings("deprecation")
    public static void setSoTimeout(int soTimeout) {
        HttpClientFactory factory = getStaticValue(HttpCommandExecutor.class, "httpClientFactory");
        if (factory == null) {
            factory = new HttpClientFactory();
        }

        DefaultHttpClient httpClient = (DefaultHttpClient) factory.getHttpClient();
        HttpParams params = httpClient.getParams();
        HttpConnectionParams.setSoTimeout(params, soTimeout);
        httpClient.setParams(params);

        setStaticValue(HttpCommandExecutor.class, "httpClientFactory", factory);
    }

    private static <T> T getStaticValue(Class<?> aClass, String fieldName) {
        Field field = null;
        Boolean isAccessible = null;
        try {
            field = aClass.getDeclaredField(fieldName);
            isAccessible = field.isAccessible();
            field.setAccessible(true);

            return (T) field.get(null);

        } catch (NoSuchFieldException e) {
            throw new RuntimeException(e);
        } catch (IllegalAccessException e) {
            throw new RuntimeException(e);
        } finally {
            if (field != null && isAccessible != null) {
                field.setAccessible(isAccessible);
            }
        }
    }

    private static void setStaticValue(Class<HttpCommandExecutor> aClass, String fieldName, Object value) {
        Field field = null;
        Boolean isAccessible = null;
        try {
            field = aClass.getDeclaredField(fieldName);
            isAccessible = field.isAccessible();
            field.setAccessible(true);

            field.set(null, value);

        } catch (NoSuchFieldException e) {
            throw new RuntimeException(e);
        } catch (IllegalAccessException e) {
            throw new RuntimeException(e);
        } finally {
            if (field != null && isAccessible != null) {
                field.setAccessible(isAccessible);
            }
        }
    }
}

Trading: Call Options - introduction

Investment oportunity

Imagine that you would meet somebody who is from the future.

And that somebody would tell you that each gold bar which
now costs 100$
will cost 115$ in a years time.

And imagine that for some reason you would believe him (as for example he looks exactly like you just 10 years older).

Now stop for a moment and think: What would you do regarding this gold situation?

Possible solution

It seems that a wise choice would be to invest all your money into gold and then sell it in a year.

You have 5,000$ in savings and the bank is willing to lend you maximum amount of 250,000$ with a generous interest rate of 5% per year.

So, let's do some calculations.

How much money do we actually have? This is a simple formula:
   our savings + borrowed money = 5,000 + 250,000 = 255,000 $

So, now the question is how many gold bars can we buy for this amount. To get this value we just divide our total amount by today's price of one gold bar:
   amount we have / today's price of gold bar = 255,000 / 100 = 2,550 gold bars

2,550 gold bars seems like a quite nice amount. But it would be good to actually know how much money we'll get for it in a year. To find it out just use this formula:
   amount of gold bars * price of gold bar in a year = 2,550 * 115 = 293,250 $
 
It seems that we have earned something. But how much actually is it?

First find out how much we have gained above our invested amount:
   earned amount - invested amount = 293,250 - 255,000 = 38,250 $

This seems like a quite nice sum but this is still not our net/clear income as we still need to pay the interest for the borrowed money which is:
   borrowed amount * interest rate percentage / 100 = 250,000 * 5 / 100 = 12,500 $

Once we pay this amount as well we'll get the clear income:
   investment return - interest for borrowed money = 38,250 - 12,500 = 25,750 $

So with our savings (5,000$) and borrowed money (250,000$) we've been able to earn 25,750$. As we return the borrowed money but keep the savings we now have:
   clear income + savings = 25,750 + 5,000 = 30,750$

This seems to be quite a nice year, don't you think?

Return percentage

Lets do some deeper analysis of how good we actually did.

At first we'll look what was the best/maximum percentage of return we could have achieved using this method.

To get this we'll find the difference of future gold price and its current price value:
   price in a year - price today = 115 - 100 = 15 $

And from this we find how big increase it actually is:
   (price difference / price today) * 100 = (15 / 100) * 100 = 15%

So 15% is the maximum we could have gained if we would use only our savings (and there would be no need to pay interest for borrowed money).
Which means that for each 100 $ we would earn 15% which is 15 $.

So if we would use just our savings we would gain: 5,000 * 15 / 100 = 750 $.

But as we were borrowing some money and we paid some interest rate, lets find out how good we actually did. For this purposes we use this calculation:
   (net income / invested amount) * 100 = (25,750 / 255,000) * 100 = 10.098 %

So lets summarize:
With using just our savings (5,000 $), we would have the highest return percentage (15%) but as only small amount would be invested, we would gained lower total income as well = 750 $.

With the borrowed money our total investment has increased rapidly (255,000 $) and although the return percentage was lower (10.098% - which is still regarded as good investment) our total income was much higher = 25,750 $.

It seems that it was a good decision to use not just our own savings but to borrow some money from bank as well.

Congratulations.

Other options - Call option

Now the question is: Can we do better than this?

Imagine that during our application for the bank loan, we would state that: I would like to buy gold bars because I expect their value to rise in a years time.

Our friendly banker would for some reason stop for a while. After a moment of deep thoughts he would tell us:
"I would have a proposal for you. Of course we will lend you the money. You're a good client as you already have 5,000 $ of your own, so I'm sure you'll be able to handle additional 250,000 $.
But why would you buy all of this gold right now, when instead I would offer you something better.

I can offer you the possibility to buy gold bars from us in a year for the same price as they costs today. So if the price of gold will be higher then today you'll earn some money. And the best thing is that if the price is lower there is no obligation for you to buy the gold bars from us, so you'll lose no more money at all. This seems really promising doesn't it? And to do so, all you have to do is sign this contract and pay the initial fee of 8 $ for each gold bar you'll be able to buy.""

Ok, so our friendly banker has offered us the possibility (but not the obligation) to buy something in the future for an agreed price. This is a contract which is normally known as 'Call Option'. The fee you have to pay for signing/entering this contract is known as 'Option Premium'.

So we have a new possibility now. But is it worth using this possibility? Let's do some math again to find it out.

First we'll find out how many 'Call options' can we actually buy:
   amount we have / option premium = 255,000 / 8 = 31,875 pieces

Now we have some amount of 'Options'. Lets imagine, that instead of buying the gold in years time for 100 $ and selling it back for 115 $ there are people who would buy our contracts for the price difference 15 $. So how much would we gain in the end:
   count of option calls * difference in gold price = 31,875 * 15 = 478,125 $

OMG, this is a horrendous sum. But remember we still need to subtract our invested amount and interest we need to pay for borrowed money to get the net/real income:
   investment return - invested amount - interest for borrowed money = 478,125 - 255,000 - 12,500 = 210,625 $

Holy macarony. This is way better than the previous 25,750 $. Lets see the actual percentage of return:
   (net income / invested amount) * 100 = (210,625 / 255,000) * 100 = 82.598 %

This seems like the best possible approach. We received much more money (210,625 $ instead of 25,750 $) as well as higher return percentage (82.598 % instead of 10.098 %).

The funny thing is that we could never achieve something like this with just buying gold on its own. Instead we were buying something that was derived from buying the gold. An option to buy the gold. Because of that we say that 'Call Option' is a 'Derivative'. There are many more 'Derivatives' that are being used, but for now just remember that 'Call Option' is one of them.

Negative scenario - possible losses explained

You might ask yourself. Why would the friendly banker ever provide us such an excellent opportunity? Aren't they only smiling on the outside but are rotten inside?

Well actually our friendly banker is nor good nor bad. All what he did is provided us the possibility to "multiply" the possible gain, but also (and this is important) the possible loss.

Let's imagine that all of the future traveling would be just a prank and the price of gold bar in years time would instead of rising by 15 $ to 115 $ actually fell by (only) 5 $ to 95 $.

How much would we actually loose?

We've already discussed few scenarios and each of them would have different losses:

a) if we would use only our savings

For 5,000 $ we could buy only 50 gold bars. In one years time the price of those gold bars would be:
   amount of gold bars * price of gold bar in a year = 50 * 95 = 4,750 $

So we would loose:
   earned amount - invested amount = 4,750 - 5,000 = -250 $

Which in percentages is a loss of:
   (clear loss / invested amount) * 100 = (250 / 5,000) * 100 = 5 %

b) if we would use savings and borrowed money

The actual price we would receive once selling the gold bars would be:
   amount of gold bars * price of gold bar in a year = 2,550 * 95 = 242,250 $

So the loss would be:
   earned amount - invested amount = 242,250 - 255,000 = -12,750 $

As we still need to pay the interest for borrowed money, our clear loss would be:
   investment difference + interest = 12,750 + 12,500 = 25,250 $

Which in percentages is a loss of:
   (clear loss / invested amount) * 100 = (25,250 / 255,000) * 100 = 9.902 %

This means that we would not even loose all of our savings, but beside this we would still need to pay back the dept of 20,250 $.

c) if we would call options with our savings and borrowed money

in this case we would actually loose everything. And by losing everything I mean we would loose our savings, borrowed money and would still need to return it plus the interest. The problem is that we used all the money to buy a promise which is now worthless. In situations before we at least had gold which was cheaper but still had SOME value and people would be willing to pay this smaller value for it. But now we only hold many papers which allow us to buy gold for higher value than we would get on the market. And nobody (with properly functioning mind) would pay us for this disadvantageous option. So in the end we would be in debt of:
   borrowed money + interest = 250,000 + 12,500 = 262,500 $

But remember, to get the net/real loss we still need to add our lost savings:
   262,500 + 5,000 = 267,500 $
 
Which in percentages is a loss of:
   (clear loss / invested amount) * 100 = (267,500 / 255,000) * 100 = 104.902 %

So as you can see, if this would be "only" a prank, it could be a very costly prank indeed.

How paying the Highest market rates can Ruin Your Company

Although it may sound absurd, paying the highest market rates can actually ruin your company or project. And this is how it goes:

You have loads of money, starting a new project and want to attract the best guys on the market.

You will make sure your interview process is tough and will offer market rates nobody else can beat (let say 200 - 300 % of market average).

You hire the best guys, plus some guys that are not the best but can talk the talk.

Project started, there is a constructive progress and everybody is happy.

People realize more and more their high (money) status and know that they could never earn that same amount anywhere else.

People start spending their money and get used to their status. Project is still constructive.

After some time somebody will realize, that he would afraid of loosing this great job and he starts to be less constructively critical (as this could maybe threaten his position).

More and more people are less and less critical as they value the earnings more than the need to say what they think should be right.

After some time the project lead/manager/main architect will get used to situation where nobody (or just few) people express concerns regarding his ideas and will actually start to believe that he is always right (also he is earning a lot of money so he must be very good as well).

After some time he will believe so much in his true that he will be annoyed when somebody will oppose him.

The leading guy will have a bad idea (everybody can have a bad idea), but nobody or just a small amount of people will point to flaws of this suggestion.

The bad idea will be implemented. And the guys that will still have the will to point out the bad concepts behind the idea are seen as a threat (to the leading guys ego and to everybody else as being befriended with them could be regarded as risk to current position).

This repeats itself few times.

Now comes the point when the guys who actually still care are becoming frustrated because they realize they can't change things if they are heading the wrong directions.

The guys that care are leaving the project as they can't handle the frustration anymore.
The guys that don't care stay as the money is good.

You're left with a main guy (or guys) with boosted ego and no self control and bunch of people that don't want to oppose bad ideas, just want to do their work, get the check and go home.

More and more bad ideas are being implemented without any constructive conversation.

Project starts to have difficulties. You hire more guys.

Those that criticize the current state are seen as threat and its slowly taken care of them (they have some bad reviews and are being fired or are assimilated and became "silent workers").

After some time you just realize that you have a project that is extremely expensive and is producing (if ever) product of questionable quality and it seemed to get wrong on many levels.

And all of this because you provided rates nobody else could beat (so only those who cared about money stayed and the reasonable guys left).

End of the story.