Hello VMmark team,
I am trying to benchmark a vSAN cluster. I started with 4 tiles. The run is being marked as non-compliant. There are quite a few exceptions reported from the weathervane application. I have two VMware clusters for the SUT and the client systems and an external SAN storage system with SSDs providing the shared iSCSI datastores for infrastructure operations. All systems are connected via two 10Gbps ethernet switches. The vSAN traffic has 2x dedicated 10Gbps ports on the SUT. There are 2x 10Gbps ports dedicated for the vMotion and iSCSI datastores. I do not see any dropped packets, re-transmits, etc., on the switches, so I am not sure if these exceptions are due to the network. Can you please advise what else could be potentially causing these issues? Thanks.
Warnings Messages::
p0 : WeathervaneAuction0 Exceptions : 5
p0 : WeathervaneElastic0 Exceptions : 1
p1 : WeathervaneAuction0 Exceptions : 2
p1 : WeathervaneElastic0 Exceptions : 231
p2 : WeathervaneAuction0 Exceptions : 5
p2 : WeathervaneElastic0 Exceptions : 446
rampdown : WeathervaneAuction0 Exceptions : 4
rampdown : WeathervaneElastic0 Exceptions : 377
p0 : WeathervaneAuction1 Exceptions : 3
p0 : WeathervaneElastic1 Exceptions : 1
p1 : WeathervaneAuction1 Exceptions : 1
p1 : WeathervaneElastic1 Exceptions : 1
p2 : WeathervaneAuction1 Exceptions : 4
p2 : WeathervaneElastic1 Exceptions : 2
rampdown : WeathervaneAuction1 Exceptions : 3
rampdown : WeathervaneElastic1 Exceptions : 1
p0 : WeathervaneAuction2 Exceptions : 5
p0 : WeathervaneElastic2 Exceptions : 149
p1 : WeathervaneAuction2 Exceptions : 6
p1 : WeathervaneElastic2 Exceptions : 373
p2 : WeathervaneAuction2 Exceptions : 5
p2 : WeathervaneElastic2 Exceptions : 348
rampdown : WeathervaneAuction2 Exceptions : 1
rampdown : WeathervaneElastic2 Exceptions : 228
p0 : WeathervaneAuction3 Exceptions : 4
p0 : WeathervaneElastic3 Exceptions : 108
p1 : WeathervaneAuction3 Exceptions : 5
p1 : WeathervaneElastic3 Exceptions : 184
p2 : WeathervaneAuction3 Exceptions : 5
p2 : WeathervaneElastic3 Exceptions : 312
rampdown : WeathervaneAuction3 Exceptions : 2
rampdown : WeathervaneElastic3 Exceptions : 218
Summary ::
Run_Is_NOT_Compliant
Turbo_Setting : 0
Number_of_Workloads_Missing : 0
Number_of_Compliance_Issues (identified by '*' or '+') : 6
Issues Found :
Tile0-weathervaneelastic-p0
Tile2-weathervaneelastic-p0
Tile2-weathervaneauction-p1
Tile2-weathervaneauction-p2
Tile3-weathervaneauction-p2
Tile3-weathervaneelastic-p2
Median_Phase : p2
I looked in the wrf files from weathervane, and I see the following sampling of exceptions:
19:26:28.029 [pool-3-thread-91] WARN c.v.w.w.common.core.Operation - Operation:run Execution Failed for GetNextBid for behavior UUID 41ad0ea8-7390-44d3-99f7-96b5ee58113d Failure Reason = com.vmware.weathervane.workloadDriver.common.exceptions.OperationFailedException: Incomplete response received when retrieving current bid for auction 30294
19:26:28.124 [pool-3-thread-91] WARN c.v.w.w.common.core.Operation - Operation:run restarting userId = 5033, operation = GetNextBid, behavior UUID 41ad0ea8-7390-44d3-99f7-96b5ee58113d Failure Reason = com.vmware.weathervane.workloadDriver.common.exceptions.OperationFailedException: Incomplete response received when retrieving current bid for auction 30294
..
19:28:07.232 [epollEventLoopGroup-3-42] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.timeout.ReadTimeoutException: null
| 200| 3733.09| 0.021| 37484| 0| 0|GetNextBid:15313/0(60/0.000/0.000), GetUserProfile:354/0(2/0.014/0.000), AddImageForItem:69/0(5/0.037/0.000), AddItem:523/0(2/0.015/0.000), Login:359/0(3/0.015/0.000), GetPurchaseHistory:339/0(2/0.039/0.000), GetImageForItem:120/0(5/0.023/0.000), GetActiveAuctions:5448/0(2/0.011/0.000), UpdateUserProfile:185/0(2/0.017/0.000), PlaceBid:1153/0(2/0.010/0.000), GetBidHistory:164/0(2/0.014/0.000), JoinAuction:2656/0(3/0.042/0.000), HomePage:344/0(2/0.021/0.000), GetItemDetail:2800/0(2/0.025/0.000), Register:0/0(2/0.000/0.000), NoOperation:0/0(9999999/0.000/0.000), Logout:345/0(3/0.014/0.000), GetCurrentItem:2520/0(3/0.015/0.000), GetAuctionDetail:2786/0(2/0.032/0.000), GetAttendanceHistory:164/0(2/0.012/0.000), LeaveAuction:1842/0(2/0.011/0.000), | Sep 28,2019 19:28:23 EDT
| Time| TP| Avg RT| Ops| Ops| Ops|Per Operation: Operation:Total/FailedRT(RT-Limit/AvgRT/AvgFailingRT)| Timestamp
| (sec)| (ops/s)| (sec)| Total| Failed| Fail RT|