https://linen.dev logo
#help
Title
# help
m

Mehmet Gul

11/29/2023, 5:21 PM
Hi In our project we were using version 2.33.2 and as tech refresh we upgraded our version to 3.3.1 We have a wiremock pod that has 4 container in it. After upgrade the behavior has changed. While running the performance test, it start throttling and causing 499 errors. I resolve the issue increasing the number of container from 4 to 10 but this is like a work around. I tried couple of parameters like: Previous setup: --async-response-enabled=true --async-response-threads=100 --container-threads=500 --disable-request-logging --no-request-journal after upgrade added params: --async-response-enabled=true --async-response-threads=100 Any helpful suggestions that I can resolve this throttling issue?
t

Tom

11/29/2023, 5:53 PM
Did you remove
--container-threads=500
in your latest deployment? If so that might be the problem.
m

Mehmet Gul

11/29/2023, 5:54 PM
No, this is the present set up. Do I need to remove it?
t

Tom

11/29/2023, 5:54 PM
No, you should keep it
m

Mehmet Gul

11/29/2023, 5:54 PM
Presently I have all these --async-response-enabled=true --async-response-threads=100 --container-threads=500 --disable-request-logging --no-request-journal
I am trying to understand what is causing throttling and need to resolve that issue.
t

Tom

11/29/2023, 5:55 PM
Shouldn’t matter but it looks like you’ve got the async-* params twice
m

Mehmet Gul

11/29/2023, 5:56 PM
sorry I wrote here by mistake while copying
t

Tom

11/29/2023, 5:57 PM
Also probably not an issue, but 500 container threads is probably too high. How many CPU cores are you allocating per container?
m

Mehmet Gul

11/29/2023, 5:58 PM
resources: requests: cpu: 250m memory: 1024Mi limits: cpu: 1000m memory: 1024Mi
this is wiremock setup
But 500 thread we were using from older version..
I am using locust performance test and sending per min 20K - 40K and 80K request tests.
t

Tom

11/29/2023, 6:00 PM
Hard to know what’s going on without more data.
m

Mehmet Gul

11/29/2023, 6:00 PM
What can I provide as more data
t

Tom

11/29/2023, 6:01 PM
I guess the CPU usage profile has shifted between versions, assuming nothing else has changed on your side
But I’d need some profiler data for before/after
e.g. from JFR
m

Mehmet Gul

11/29/2023, 6:01 PM
When I move back to older version all works fine.
t

Tom

11/29/2023, 6:01 PM
Is the WireMock version the only variable?
m

Mehmet Gul

11/29/2023, 6:02 PM
yes, same set up with older version works fine and when I change the newer version, this hapens
Hi @Tom I did binary search on versions to see where the issue starts. I have seen that beta-10 has no issue while running the locust performance testing. But in beta-11 the CPU throttling issue starts. It seams due to some change in beta-11 causing this issue. https://github.com/wiremock/wiremock/releases/tag/3.0.0-beta-11 I tried to check the code overall but nothing I see stands out. But you know better and it might ring any bell for you. I tested many versions and beta-11 is the starting point for the issue. Do you mind checking the beta-11 to see what might cause this issue. thanks
t

Tom

12/04/2023, 4:08 PM
I can't think of anything off the top of my head that obviously would have changed the performance characteristics between those releases so if you could grab and share some profiler output, ideally before and after the version upgrade that would help get to the root cause.
m

Mehmet Gul

12/04/2023, 4:39 PM
I will check those details and will try to provide you more data but do you suggest which options should we enable for JFR for you that would be helpful for you? Because with 100's of thousands of requests, I’m not sure if the JFR data would be humungous.
t

Tom

12/04/2023, 6:24 PM
I don’t think we need tons of data, probably only a few 10s of requests for each version
If each JFR recording shows a significantly different profile when you look at them then they’re probably useful
m

Mehmet Gul

12/04/2023, 7:06 PM
Ok, I will work on it and will get back to you Tom. thanks
Hi @Tom Here is the JFR for 2 version. Version 2.33.2 which is perfectly working. Version 3.33.1 the latest version. I tried to put as much info as possible for you to analyze the issue. In zip file you will find: 1- Dynatrace CPU details 2- Locust perf test results 3- JFR files 4- Text file with some details like start, end time number of request details. 5- Configuration file that we use for wiremock
t

Tom

12/18/2023, 8:30 PM
Thank you for taking the time to share this. I’m maxed out trying to get end-of-year stuff sorted at the moment but I’ll try to take a look over the holidays or early next year.
m

Mehmet Gul

12/19/2023, 1:24 PM
@Tom Do you want me to create a GitHub issue and put those details over there ?
t

Tom

12/19/2023, 1:29 PM
Yes please, that would be really helpful
m

Mehmet Gul

12/19/2023, 1:31 PM
Sure, Thanks
👍 1
I created the issue in GitHub and here it is: https://github.com/wiremock/wiremock/issues/2541
🙏 1
🙌 1
@Tom Hi Tom, just following up, any near future plan to look at this issue?
t

Tom

01/15/2024, 3:28 PM
Yes, I haven't forgotten. Just spread a bit thin at the moment as our head of community recently left the company.
m

Mehmet Gul

01/15/2024, 3:29 PM
Thanks, Tom
@Tom Hi Tom, quick question. When you guys start working on issues, do we get any notification that the issue is under review or something like that or I need to check with you guys to see if it is in progress. We would like to upgrade our wiremock version but due to performance issue keeping it in backlog. May be you can do a valentine’s day surprise and move it to in progress 🙂
t

Tom

02/13/2024, 8:38 PM
If there's any progress worth communicating then we'll comment on the GH issue so you can subscribe to that. But it's open source I'm afraid so it's best effort prioritisation. If this is urgent for you, finding the problem and creating a fix PR would move things forward more quickly.
@Mehmet Gul I’ve had a quick scan of this and have 2 hypotheses - 1) large numbers of sub-events being added on each request (fixed in 3.4.1), 2) redundant matching happening on stubs e.g. trying to match body despite the method/URL not being a match. One question - are your tests producing any unmatched requests i.e. where you get the diff report in the response?
m

Mehmet Gul

02/23/2024, 2:25 PM
Hi @Tom Thanks for follow up. For #2, we aren’t sending any requests to wiremock that are not matched by URI. We only call /mock/auth/api1 through /mock/auth/api200 which each have a 1000ms delay.
t

Tom

02/23/2024, 2:26 PM
I’ve actually confirmed 2) now and am about to push a fix. Would be really helpful if you could run your tests against my local build. If I sent you a standalone JAR, would you be able to do that?
m

Mehmet Gul

02/23/2024, 2:26 PM
sure, I can try.
t

Tom

02/23/2024, 2:28 PM
OK, one sec…running a couple more tests then I’ll send the JAR your way
m

Mehmet Gul

02/23/2024, 2:28 PM
sure, thanks
t

Tom

02/23/2024, 2:31 PM
wiremock-standalone-3.4.2.jar
m

Mehmet Gul

02/23/2024, 2:32 PM
Normally we are running wiremock in containers, I will check with our developers and test it. I will let you know once tested.
t

Tom

02/23/2024, 2:33 PM
OK, let me know if it’s an issue
m

Mehmet Gul

02/23/2024, 2:33 PM
sure
Hi Tom I have discussed with our developer team and they said we cannot do that. I can try the version 3.4.1 and update you on it. And also the 2nd option is not applicable for as we are not sending any request to wiremock that are nt matched by URI.
t

Tom

02/23/2024, 5:05 PM
I’ve just released 3.4.2 anyway so the Docker image will be available shortly. I wouldn’t bother trying 3.4.1 as it doesn’t have my fix in it.
m

Mehmet Gul

02/23/2024, 5:05 PM
I can try 3.4.2 directly once the docker image is available.
t

Tom

02/23/2024, 5:13 PM
That’s published now
m

Mehmet Gul

02/26/2024, 10:28 PM
Hi @Tom I tested the 3.4.2-1 version. Here is my observation: There is improvement compare to previous tests. 20K req/per min works fine. 40K req/per min has issue but after 15min on total 30 min testing. 80K req/per min was total fail. I am attaching the performance test results screenshot and CPU Throttling details.
t

Tom

02/27/2024, 9:28 AM
I’m not sure what to make of these results. Do you have any JFR traces you can share? Also, what would really help is if you can share the stubs you’re using to test (happy to receive privately via email if you don’t want to send them via Slack).
m

Mehmet Gul

02/27/2024, 6:38 PM
Hi @Tom I DM the details to you asked. I hope that helps.
2 Views