This message was deleted.

I’ve wrote a small Proof Of Concept API, with Spring MCV + WebClient, and my goal is to load test it in several situations. This POC is relying on calls made to a Wiremock thirdparty API In my initial nominal case, each calls to the poc API triggers 10 sequential calls to the Wiremock backend. In such context, everything is working fine. In my second test, I’m trying to increase the throughput by performing the 10 calls in parallel . In that case, the poc complains several times about

java.net.SocketException: Connection reset

So I guess that Wiremock drops certain of the connexions, but I can’t figure out why, and nothing is logged about it so far. Notice that I’m starting Wiremock with the following options :

--port 8081 --no-request-journal --disable-request-logging --async-response-enabled=true --verbose=true

Hi Benoit, can you share your test project?

Hey Tom !

That would be very helpful. With these kinds of stability/performance issues I find it’s generally impossible to diagnose based on bits of 2nd hand info. But if I’ve got something I can run that demonstrates the issue I can usually find the issue.

I’ve been running the standalone version since monday, downloaded from the Wiremock website : ~/Downloads/wiremock-standalone-3.4.1.jar And this morning I’ve git cloned and build the latest version from the github repo

Are you just running it on your laptop, or is it deployed somewhere?

I’m running a local Prometheus server

--print-all-network-traffic

at startup will give you a reasonable amount of detail

fully local run, as I don’t want to bug my company’s API Management layer, even the preproduction one

OK, so the caller and WireMock are communicating over your

localhost

?

to be honest I’m not really good regarding network subject : sometimes I get confused regarding different behavior between using localhost and 127.0.0.1 ... 🫥 but yes, i’ve configures everything over localhost as much as I can.

The main thing I wanted to establish is that you’re not connecting over e.g. a Docker network or VM environment’s network, which it sounds like you’re not. Other thing off the top of my head to look at is the underlying HTTP client being used by Spring. There have been some in the past that exhibited the kind of problem you’ve described under load - older versions of Netty for instance, although I’ve not seen this for a while.

Ok, I’ll try that also

I’d be surprised if it was this issue based on the load levels you’ve been using.

same trouble whenever simply adding the

--container-threads=200

option on the downloaded wiremock-standalone-3.4.1.jar

That’s irritating. I use

tcpdump

in cases like this

yep, used it once or twice for really complex situations, but as I’m really not experienced with ISO layers, it’s always a pain in the ass for me to understand what Wireshark is spitting at me whenever I need to analyse it 😄 I’m more into deep dive debugging in general. But in that particular situation, don’t know where to place breakpoints, especially due to the fact it seems to be impossible to do for Intellij in the Jetty layers ... I guess it’s due to having Jetty embedded into the Wiremock jar, but I’m not 100% certain. Anyway, I’m working on building an anonymized version of my poc to send it to you. It will be (hopefully) more straightforward

I'm on a work trip at the moment so haven't had a chance to look but I think one of my team will pick it up

Hi Noticed 🙂 Thanks for the feedback Enjoy your trip !

Hi, I am having trouble with high memory usage in performance testing. Ok this may be normal but what is weird is that memory usage goes up to a value (for example 3500 mb) and never drop down even if performance testing is finished or there is no incoming request to the server. It keeps the same memory usage all the time. In this case, I will need to restart server all the time. And this is a manuel process and vulnerable to connection issues when it is forgotten. Could you please help? What can I do?

@Orçun BALCILAR: Hi You can maybe have things getting better by using the Wiremock parameters I’ve mentioned in my second message in this thread ?

I've tried these 3 parameters -> --no-request-journal --disable-request-logging --async-response-enabled=true many times. But it didn't work. Do you have the link?

I have the following ones regarding that matter : https://github.com/Rbillon59/wiremock-loadtest https://groups.google.com/g/wiremock-user/c/2TsjGAIU350 If none of them work, only memory profiling could help you spotting what is consuming so much. But it would takes to git clone the whole Wiremock project, manage to have it building, and run it with a right memory profiling tool, such as the one from Intellij, or MAT, which can quickly get quite complex

Thank you @Benoit LEFEVRE -CAMPUS-. I've figured out that memory consumption value goes up only once and stays at that level wheher load test is done again or not. So perhaps it is jetty/threads that keeps heap memory.

@Orçun BALCILAR: can’t remember during which technical presentation I’ve heard about it, but either Netty or Jetty is doing very low level memory management on is own, I.E. asking memory frames to JRE low level API (some that shouldn’t be used anymore) and that can lead to strange memory behavior. But it’s pure speculation from me, as I’m not familiar at all with those two techs anyway. I guess @Tom should be able to tell us more about it 🙂

I wouldn’t like to speculate without some data. It sounds like it’s not any of the obvious things like an ever-growing request journal. I suggest grabbing a thread dump and a heap dump while the server is idle.

yup, using mat on a dump should be a good first guess to understand what is clogging the memory that way

I will create a repo to reproduce the issue and will share with you. If I am able to analyze memory, I will share results too.

What memory measurement is this? Is it the operating system’s view of memory committed to the JVM process or the JVM’s measure of heap size, or something else?

operating system’s view of memory committed to the JVM process on task manager window

Java doesn’t return memory to the OS by default so this isn’t surprising. I suspect most of that memory is unallocated heap inside the JVM, but you can confirm this via a thread/heap dump. If you want it to consume less OS memory you can set a lower -Xmx value on JVM startup. You probably don’t need as much as 3.5G and it’ll just mean you get smaller, more frequent GCs.

Ah sorry, I didn't know -> Java doesn’t return memory to the OS by default so this isn’t surprising. Thank you for the answer. Another point that I came across is that average response time is much bigger (500 ms) when I use response custom handlebars I coded and built in handlebars. But when I switched to response transformer, average response time gets too low (for example 2 ms). So I preferred response transformer to get low response time in performance testing.

So your custom code was faster, or the built-in response templating?

my custom response transformer code ( -> public class ...ResponseTransformer implements ResponseDefinitionTransformerV2) is much faster than my custom response templating ( -> public class XmlPartTemplate implements TemplateHelperProviderExtension)

Are you running the latest version of WireMock? There was a PR quite recently from one of our team that significantly improved the performance of larger templates.

yes wiremock 3.5.2

Hi @Tom. How are you doing ? I understand that the problematic I’ve reported here is far far from being a top priority, but on the other hand I’d rather not to loose the (tiny) effort I’ve invested in creating the reproduction case Do you think it’s a good alternative to turn it into an issue on Wiremock github ?

Hi @Benoit LEFEVRE -CAMPUS- From my experiences I’ve figured out that fine tuning is a key point in scaling wiremock server and achieving performance. What I mean is that you need to fine tune jetty connector threads and acceptors and queue size parameters depending on what tps you are planning to create. If you like, I can share a configuration example.

@Orçun BALCILAR: sure, could help seeing how to tune it the right way. But a question first : did you also encountered the

Connection reset

problematic, and solved it with Jetty parameters tuning ? Or are we maybe talking of distinct problematics ? I’m not certain anymore, as the subject start to be a bit old for my tiny memory 🙂

I am not sure but most probably I encountered. I can reproduce and make sure I am getting or not. For about 1500 tps load, I’ve used 400 connector, 200 acceptor, 1000 queue size configuration along with disable request logging. As an extra info, I also encountered a performance issue on custom and built in handlebars combined. I needed to replace it a custom whole response transformer extension to meet my average response time expectation and 100 percent success rate.