Testing Misc

log4j https://stackoverflow.com/questions/35133965/can-r-pattern-in-log4j-2-be-correctly-used-in-combination-with-java-application

%r represents the number of milliseconds since the JVM started, not necessarily when the Layout was created. The value is calculated by calling ManagementFactory.getRuntimeMXBean().getStartTime(); when the pattern converter is created and then subtracting the event timestamp from the start time value for each event. Given that the start time never changes this value should grow over time as you are describing.

Log4j doesn't have any way to get the time the request was started. You could capture that in a ThreadContext value and then create your own pattern converter to use that value as the value to subtract from the current system time.

https://stackoverflow.com/questions/15020747/how-should-i-use-org-mockito-additionalmatchers-gt

You should use Hamcrest's greaterThan for this case. gt is for verifying arguments of method calls in mock objects:

    private List<Integer> list = Mockito.mock(List.class);

    @Test
    public void testGreaterThan() throws Exception {
        assertThat(17, is(org.hamcrest.Matchers.greaterThan(10)));

        list.add(17);
        verify(list).add(org.mockito.AdditionalMatchers.gt(10));
    }

https://github.com/junit-team/junit4/wiki/timeout-for-tests
This is implemented by running the test method in a separate thread. If the test runs longer than the allotted timeout, the test will fail and JUnit will interrupt the thread running the test. If a test times out while executing an interruptible operation, the thread running the test will exit (if the test is in an infinite loop, the thread running the test will run forever, while other tests continue to execute).

The timeout specified in the Timeout rule applies to the entire test fixture, including any @Before or @After methods. If the test method is in an infinite loop (or is otherwise not responsive to interrupts) then @After methods will not be called.
https://garygregory.wordpress.com/2011/09/09/multi-threaded-unit-testing/

Callable<Long> task = new Callable<Long>() {

            @Override

            public Long call() {

                return domainObject.nextId();

            }

        };

        List<Callable<Long>> tasks = Collections.nCopies(threadCount, task);

        ExecutorService executorService = Executors.newFixedThreadPool(threadCount);

        List<Future<Long>> futures = executorService.invokeAll(tasks);

        List<Long> resultList = new ArrayList<Long>(futures.size());

        // Check for exceptions

        for (Future<Long> future : futures) {

            // Throws an exception if an exception was thrown by the task.

            resultList.add(future.get());

        }

https://www.alexecollins.com/5-tips-unit-testing-threaded-code/
Tip 1 - Life-cycle Manage Your Objects
Object that have a managed life-cycle are are easier to test, the life-cycle allows for set-up and tear-down, which means you can clean-up after your test and no spurious threads are lying around to pollute other tests.

    private ExecutorService executorService;

    public void start() {
        executorService = Executors.newSingleThreadExecutor();
    }

    public void stop() {
        executorService.shutdown();
    }
}

Tip 2 - Set a Timeout on Your Tests
Bugs in code (as you'll see below) can result in a multi-threaded test never completing, as (for example) you're waiting on some flag that never gets set. JUnit lets you set a timeout on your test.

@Test(timeout = 100) // in case we never get a notification

Tip 3 - Run Tasks in the Same Thread as Your Test
Typically you'll have an object that runs tasks in a thread pool. This means that your unit test might have to wait for the task to complete, but you're not able to know when it would complete. You might guess,

A trick is to make the task run synchronously, i.e. in the same thread as the test. Here this can be achieved by injecting the executor:

Then you can have use a synchronous executor service (similar in concept to a SynchronousQueue) to test: An updated test that doesn't need to sleep:

Tip 4 - Extract the Work from the Threading
If your thread is waiting for an event, or a time before it does any work, extract the work to its own method and call it directly.

Tip 5 - Notify State Change via Events
An alternative to the previous two tips is to use a notification system, so your test can listen to the threaded object.

    @Override
    public void update(final Observable o, final Object arg) {
        assert o == sut;
        updateLatch.countDown();
    }

Pros:

1. Creates useful code for listening to the object.
2. Can take advantage of existing notification code, which makes it a good choice where that already exists.
3. Is more flexible, can apply to both tasks and process orientated code.
4. It is more cohesive than extracting the work.

Cons:

1. Listener code can be complex and introduce its own problems, creating additional production code that ought to be tested.
2. De-couples submission from notification.
3. Requires you to deal with the scenario that no notification is sent (e.g. due to bug).
4. Test code can be quite verbose and therefore prone to having bugs

https://github.com/randomizedtesting/randomizedtesting/wiki
@RunWith(RandomizedRunner.class)
@Seed("the-seed")
@com.carrotsearch.randomizedtesting.annotations.Repeat(iterations = 5)


https://www.codeaffine.com/2016/04/06/replace-rules-in-junit5/

The test execution follows a certain life cycle. And each phase of that life cycle that can be extended is represented by an interface. Extensions can express interest in certain phases in that they implement the corresponding interface(s).

With the ExtendWith annotation a test method or class can express that it requires a certain extension at runtime. All extensions have a common super interface: ExtensionPoint. The type hierarchy of ExtensionPoint lists all places that extension currently can hook in.

@ExtendWith(TemporaryFolderExtension.class)

https://stackoverflow.com/questions/13489388/how-does-junit-rule-work

Rules are used to add additional functionality which applies to all tests within a test class, but in a more generic way.

For instance, ExternalResource executes code before and after a test method, without having to use @Before and @After. Using an ExternalResource rather than @Before and @After gives opportunities for better code reuse; the same rule can be used from two different test classes.

https://stackoverflow.com/questions/1835523/is-there-a-way-to-make-eclipse-run-a-junit-test-multiple-times-until-failure

It is possible to implement such an loop with TestRules (since JUnit 4.9)

A very simple implementation that runs every Test 10 times:

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class SimpleRepeatRule implements TestRule {

    private static class SimpleRepeatStatement extends Statement {

        private final Statement statement;

        private SimpleRepeatStatement(Statement statement) {
            this.statement = statement;
        }

        @Override
        public void evaluate() throws Throwable {
            for (int i = 0; i < 10; i++) {
                statement.evaluate();
            }
        }
    }

    @Override
    public Statement apply(Statement statement, Description description) {
        return new SimpleRepeatStatement(statement);
    }
}

usage:

public class Run10TimesTest {

   @Rule
   public SimpleRepeatRule repeatRule = new SimpleRepeatRule();

   @Test
   public void myTest(){...}
}

https://www.codeaffine.com/2013/04/10/running-junit-tests-repeatedly-without-loops/
public class RepeatRule implements TestRule {
 
  @Retention( RetentionPolicy.RUNTIME )
  @Target( {
    java.lang.annotation.ElementType.METHOD
  } )
  public @interface Repeat {
    public abstract int times();
  }

  private static class RepeatStatement extends Statement {

    private final int times;
    private final Statement statement;

    private RepeatStatement( int times, Statement statement ) {
      this.times = times;
      this.statement = statement;
    }

    @Override
    public void evaluate() throws Throwable {
      for( int i = 0; i < times; i++ ) {
        statement.evaluate();
      }
    }
  }

  @Override
  public Statement apply(
    Statement statement, Description description )
  {
    Statement result = statement;
    Repeat repeat = description.getAnnotation( Repeat.class );
    if( repeat != null ) {
      int times = repeat.times();
      result = new RepeatStatement( times, statement );
    }
    return result;
  }
}

https://stackoverflow.com/questions/15904584/mockito-gives-unfinishedverificationexception-when-it-seems-ok/31085466

This might also be caused if you try to verify a method which expects primitive arguments with any():

For example, if our method has this signature:

method(long l, String s);

And you try to verify it like this, it will fail with aforementioned message:

verify(service).method(any(), anyString());

Change it to anyLong() and it will work:

verify(service).method(anyLong(), anyString());

https://stackoverflow.com/questions/24431427/multiple-runwith-statements-in-junit

As of JUnit 4.7 and Mockito 1.10.17, this functionality is built in; there is an org.mockito.junit.MockitoRule class. You can simply import it and add the line

@Rule public MockitoRule mockitoRule = MockitoJUnit.rule();

For older versions of Mockito (down to 1.10.5 it seems), you have to use: @Rule public MockitoJUnitRule mockito = new MockitoJUnitRule(this);

https://www.mkyong.com/unittest/junit-4-tutorial-2-expected-exception-test/

        thrown.expectMessage(is("Name is empty!"));

        //test detail
        thrown.expect(hasProperty("errCode"));  //make sure getters n setters are defined.
        thrown.expect(hasProperty("errCode", is(666)));

https://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/
1. Testing algorithms together with coordinators. 

2. Mocking too much. Perhaps the greatest benefit of unit tests is that they force you to write code that can be tested in isolation. In other words, your code becomes modular. When you mock the whole world around your objects, there is nothing that forces you to separate the parts. You end up with code where you can’t create anything in isolation – it is all tangled together. From a recent tweet by Bill Wake:  “It’s ironic – the more powerful the mocking framework, the less pressure you feel to improve your design.”

https://henrikwarne.com/2014/02/09/unit-testing-private-methods/

How can you unit test private methods? If you google this question, you find several different suggestions: test them indirectly, extract them into their own class and make them public there, or use reflection to test them. All these solutions have flaws. My preference is to simply remove the private modifier and make the method package private.

Either the private method does something interesting, and then it should be unit tested in its own right, or it doesn’t do anything interesting, and then it doesn’t need to be unit tested at all.

Furthermore, this doesn’t work well if you practice Test Driven Development (TDD). If you take a bottom up approach, and develop and test the building blocks before putting them together, you often don’t have the public method ready when you are developing the helper methods. Thus you don’t get the benefits of testing while developing

https://github.com/typicode/json-server
$ npm install -g json-server

$ json-server --watch db.json

https://octoperf.com/blog/2015/06/10/top-load-testing-tools/
JMeter
Gatling tool
Locust

The grinder is sometimes in competition with JMeter in software companies. Developers usually like The Grinder because it’s devopsfriendly, to write test plans using code instead of a GUI. This tool is hosted by SourceForge.

Tsung is a multi-protocol, distributed stress testing tool. It is developed in Erlang, an open-source language made by Ericsson for building robust fault-tolerant distributed applications.

Supported protocols are HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP and Jabber/XMPP servers. Tsung purpose, like any other load and stress testing software, is to simulate users in order to test the scalability and performance of IP based client/server applications.

It can be distributed on several client machines and is able to simulate hundreds of thousands of virtual users concurrently.

randomizedtesting.RandomizedRunner
https://github.com/randomizedtesting/randomizedtesting/wiki/Core-Concepts

The main idea is this: a test case that runs in an exactly identical way every time it is run only covers that single execution path. Such tests are very good for verifying if any changes of behavior have happened when new code has been introduced (regression testing) or to assert on corner cases. These "fixed" tests do not bring any new insight into how the program behaves for previously unseen combinations of input arguments, components or environmental settings. And because for complex (or any) software such interactions are hard to predict in advance (think those buffer underruns, null pointers, etc.) running your tests on as many different input combinations as possible should over time increase the confidence that the software is robust and reliable.

The question how to implement the above concept of "different execution every time" and how to assert on conditions in such case can be solved in many ways. RandomizedRunner provides an implementation of java.util.Random which is initialized with a random seed that is reported (injected into a stack trace) in case of a test failure. So if a test fails it should be, at least theoretically, repeatable if started from the same seed.

In a randomized test case, the execution will be different every time. For the add method we may randomize the arguments (within their contract bounds) and verify if the outcome satisfies some conditions. For this example, let's say the result of adding two non-negative integers shouldn't be smaller than any of the arguments:

@Test
public void randomizedTesting() {
  // Here we pick two positive integers. Note superclass utility methods.
  int a = randomIntBetween(0, Integer.MAX_VALUE);
  int b = randomIntBetween(0, Integer.MAX_VALUE);
  int result = Adder.add(a, b);
  assertTrue(result + " < (" + a + " or " + b + ")?", result >= a && result >= b);
} 

This test passes most of the time, but occasionally it will fail due to integer overflow.

Once a failing execution has been caught it's easy to repeat it (that's the whole point!). Note the first line of the stack trace, it contains the master randomization seed picked for the execution: 2300CE9BBBCFF4C8:573D00C2ABB4AD89. The first number if the "master" seed used in static suite context (class initializers, @BeforeClass and @AfterClass hooks), the second number is the seed derived from the master and used in a test context. To repeat the exact same failing execution we could either override seeds using system properties, as in:

-Drt.seed=2300CE9BBBCFF4C8:573D00C2ABB4AD89

or we could annotate the class/method in question and fix the seed to a particular value; for instance by adding an annotation to the class:

@Seed("2300CE9BBBCFF4C8:573D00C2ABB4AD89")

After doing so, we would be set for a debugging session to see what the cause of the problem was. The above example is part of a walk-through tutorial available in progressive difficulty.

https://www.mkyong.com/unittest/hamcrest-how-to-assertthat-check-null-value/

        //true, check null
        assertThat(null, is(nullValue()));
        //true, check not null
        assertThat("a", is(notNullValue()));

http://locust.io/
pip install locustio
or
brew install libevent

Locust requires Python 2.7+. It is not currently compatible with Python 3.x.


locustfile.pychttps://github.com/junit-team/junit4/wiki/exception-testing

Alternatively, use the ExpectedException rule. This rule lets you indicate not only what exception you are expecting, but also the exception message you are expecting:

@Rule
public ExpectedException thrown = ExpectedException.none();

@Test
public void shouldTestExceptionMessage() throws IndexOutOfBoundsException {
    List<Object> list = new ArrayList<Object>();

    thrown.expect(IndexOutOfBoundsException.class);
    thrown.expectMessage("Index: 0, Size: 0");
    list.get(0); // execution will never get past this line
}

@Test(expected = IndexOutOfBoundsException.class) 
public void empty() { 
     new ArrayList<Object>().get(0); 
}

@Test
public void testExceptionMessage() {
    try {
        new ArrayList<Object>().get(0);
        fail("Expected an IndexOutOfBoundsException to be thrown");
    } catch (IndexOutOfBoundsException anIndexOutOfBoundsException) {
        assertThat(anIndexOutOfBoundsException.getMessage(), is("Index: 0, Size: 0"));
    }
}

http://maven.apache.org/components/surefire/maven-failsafe-plugin/examples/single-test.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html

During development, you may run a single test class repeatedly. To run this through Maven, set the test property to a specific test case.

mvn -Dtest=TestCircle test

Spring + JUnit
http://docs.spring.io/spring/docs/current/spring-framework-reference/html/integration-testing.html
https://github.com/CodeSolid/tutorials/tree/master/SpringJUnitIntro
https://www.mkyong.com/unittest/junit-spring-integration-example/
https://examples.javacodegeeks.com/core-java/junit/spring-junit-test-example/
JUnit 4.11doesn't work with Spring Unit test framework.

1. JUnit 4.12
2. Hamcrest 1.3
3. Spring 4.3.0.RELEASE

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = {AppConfig.class})
public class MachineLearningTest {}

Eclipse JUnit - possible causes of seeing “initializationError” in Eclipse window
http://stackoverflow.com/questions/15190699/eclipse-junit-possible-causes-of-seeing-initializationerror-in-eclipse-windo

You've probably got one of two problems:

1) You're using JUnit 4.11, which doesn't include hamcrest. Add the hamcrest 1.3 library to your classpath.

2) You've got hamcrest 1.3 on your classpath, but you've got another version of either junit or hamcrest on your classpath.

For background, junit pre 4.11 included a cut down version of hamcrest 1.1. 4.11 removed these classes.

TestNG
http://www.mkyong.com/unittest/testng-tutorial-5-suite-test/

https://examples.javacodegeeks.com/enterprise-java/testng/testng-beforesuite-example/

A @BeforeSuite annotated method represents an event before the suite starts, so all the @BeforeSuite methods will be executed before the first test declared within the test element is invoked.

When we run testng.xml, we see the order in which the annotated methods are fired. The very first methods that run are the @BeforeSuite methods. Since each of the test classes have one @BeforeSuite, both are consecutively run and then only other annotated methods are fired starting with @BeforeTest method.

http://stackoverflow.com/questions/9003754/implementing-testng-suite-programmatically-and-executing-it-via-maven-surefire

 @BeforeSuite
 public static void TestSuite() {
    TestNG testNG = new TestNG();

    List<Class> listnerClasses = new ArrayList<Class>();
    List<String> suiteNameList = new ArrayList<String>();
    Class[] classList = new Class[]{
            ManagementClient.class,
            ProcessManagementClient.class,
            DeployClient.class,
            RedeployClient.class};
    listnerClasses.add(org.wso2.platform.test.core.PlatformTestManager.class);
    listnerClasses.add(org.wso2.platform.test.core.PlatformSuiteManager.class);        
    testNG.setDefaultSuiteName("suite");
    testNG.setListenerClasses(listnerClasses);
    testNG.setTestClasses(classList);
    testNG.run();
}

Junit

https://github.com/junit-team/junit4/wiki/Aggregating-tests-in-suites

The class below is a placeholder for the suite annotations, no other implementation is required. Note the @RunWith annotation, which specifies that the JUnit 4 test runner to use isorg.junit.runners.Suite for running this particular test class. This works in conjunction with the @Suite annotation, which tells the Suite runner which test classes to include in this suite and in which order.

@RunWith(Suite.class)
@Suite.SuiteClasses({
  TestFeatureLogin.class,
  TestFeatureLogout.class,
  TestFeatureNavigate.class,
  TestFeatureUpdate.class
})

public class FeatureTestSuite {
  // the class remains empty,
  // used only as a holder for the above annotations
}

https://github.com/thoughtbot/guides/tree/master/best-practices

• Avoid any_instance in rspec-mocks and mocha. Prefer dependency injection.
• Avoid its, specify, and before in RSpec.
• Avoid let (or let!) in RSpec. Prefer extracting helper methods, but do not re-implement the functionality of let.Example.
• Avoid using subject explicitly inside of an RSpec it block. Example.
• Avoid using instance variables in tests.
• Disable real HTTP requests to external services with WebMock.disable_net_connect!.
• Don't test private methods.
• Test background jobs with a Delayed::Job matcher.
• Use stubs and spies (not mocks) in isolated tests.
• Use a single level of abstraction within scenarios.
• Use an it example or test method for each execution path through the method.
• Use assertions about state for incoming messages.
• Use stubs and spies to assert you sent outgoing messages.
• Use a Fake to stub requests to external services.
• Use integration tests to execute the entire app.
• Use non-SUT methods in expectations when possible.

https://robots.thoughtbot.com/fake-it

http://readwrite.com/2008/08/14/12_unit_testing_tips_for_software_engineers/
5. Set up Clean Environment for Each Test
6. Use Mock Objects To Test Effectively
7. Refactor Tests When You Refactor the Code
8. Write Tests Before Fixing a Bug
http://howtodoinjava.com/best-practices/unit-testing-best-practices-junit-reference-guide/

• Unit tests as opposed to say Integration tests, are usually meant to test a single Class.
• Any class dependencies therefore should be removed from the equation (so that if a class dependency is buggy, the unit test won't be affected by it), which is done by mocking them away.
• Mocks are "fake" objects, copying all the public methods / members of the original class, while returning "default" values such as null for an object, or 0 for an int.
• Mocks can be told how they should behave, e.g. what a method should return when called on the mock. This is done with `when/then`.
• Mocks allow you to test your code's interaction with a dependency, by mocking the dependency and using `verify`.
• If the class under question is initializing a dependency directly (e.g calling `new Dependency()`), you won't be able to easily test interactions with that dependency. One easy way to fix that is to not initialize that dependency in the class, but rather receive it as a constructor argument, which then can be mocked in the test.
• One way to make sure a test is good, meaning it actually tests what it's supposed to test, is to momentarily comment out / edit the original code under test in a way that should make the test fail, and make sure the test indeed fails, in an appropraite manner.
• Naming unit tests properly is important and pretty easy, simply describe what is being tested, e.g. "throwsAnInvalidArgumentExceptionWhenIdIsIllegal"
• Ideally each test should only have one way to fail, meaning it tests a single thing

http://howtodoinjava.com/best-practices/unit-testing-best-practices-junit-reference-guide/
 Test only one code unit at a time
Don’t make unnecessary assertions
Make each test independent to all the others
Mock out all external services and state
Don’t unit-test configuration settings
Name your unit tests clearly and consistently

You must name your testcases on what they actually do and test. Testcase naming convention which uses class names and method names for testcases name, is never a good idea. Every time you change the method name or class name, you will end up updating a lot of test cases as well.

But, if your test cases names are logical i.e. based on operations then you will need almost no modification because most possibly application logic will remain same.

E.g. Test case names should be like:

1) TestCreateEmployee_NullId_ShouldThrowException
2) TestCreateEmployee_NegativeId_ShouldThrowException
3) TestCreateEmployee_DuplicateId_ShouldThrowException
4) TestCreateEmployee_ValidId_ShouldPass

Write tests for methods that have the fewest dependencies first, and work your way up
All methods, regardless of visibility, should have appropriate unit tests
Aim for each unit test method to perform exactly one assertion
Create unit tests that target exceptions
@Test(expected=SomeDomainSpecificException.SubException.class)
Use the most appropriate assertion methods.
Put assertion parameters in the proper order

Assert methods takes usually two parameters. One is expected value and second is original value. Pass them in sequence as they are needed. This will help in correct message parsing if something goes wrong.

Ensure that test code is separated from production code
Do not print anything out in unit tests
Do not use static members in a test class
Do not write your own catch blocks that exist only to fail a test
Do not rely on indirect testing
Integrate Testcases with build script
Do not skip unit tests
Capture results using the XML formatter

http://readyapi.smartbear.com/testserver/howto/text/datadriven/files

https://alexecollins.com/5-tips-unit-testing-threaded-code/

@Test(timeout = 100) // in case we never get a notification

Tip 3 - Run Tasks in the Same Thread as Your Test

Typically you'll have an object that runs tasks in a thread pool. This means that your unit test might have to wait for the task to complete, but you're not able to know when it would complete. You might guess, for example:

    private final AtomicLong foo = new AtomicLong();
...
    public void incr() {
        executorService.submit(new Runnable() {
            @Override
            public void run() {
                foo.incrementAndGet();
            }
        });
    }
...
    public long get() {
        return foo.get();
    }
}

Consider this test:

    private Foo sut; // system under test

    @Before
    public void setUp() throws Exception {
        sut = new Foo();
        sut.start();
    }

    @After
    public void tearDown() throws Exception {
        sut.stop();
    }

    @Test
    public void testGivenFooWhenIncrementGetOne() throws Exception {
        sut.incr();
        Thread.sleep(1000); // yuk - a slow test - don't do this
        assertEquals("foo", 1, sut.get());
    }
}




An updated test that doesn't need to sleep:

    private Foo sut; // system under test
    private ExecutorService executorService;

    @Before
    public void setUp() throws Exception {
        executorService = new SynchronousExecutorService();
        sut = new Foo(executorService);
        sut.start();
    }

    @After
    public void tearDown() throws Exception {
        sut.stop();
        executorService.shutdown();
    }

    @Test
    public void testGivenFooWhenIncrementGetOne() throws Exception {
        sut.incr();
        assertEquals("foo", 1, sut.get());
    }
}

Tip 4 - Extract the Work from the Threading