Re: How to create Utilities method allow serizalization and deserialization request payload and response payload.

To reiterate, if performance isn't important, you will likely be far far better off writing a quick screen scraper (such as with selenium, etc). Even if it has to load the page, read 10 rows at a time, click next, wait 3s to load, repeat, it will probably take longer to write than to run.

If you have a specific question about part of the full payload, you should share the details you've worked out so far and the part of the payload you have the question about, but it isn't going to make sense to share the full contents on the mailing list and have us give you a strategy (as it will probably just be what has already been discussed). You may not have all of the details, but you have a lot more than we do, and you can very likely us that context to work out what the format is - consider comparing your guess of what the data is with what you see in the UI.

The order of the data is what the linked post/document suggests - read the payload backwards starting just before the string array. Nothing is jumbled, the order you read values (in reverse) is the order that the data was written, starting from the list that holds the objects. The request order isn't quite the same (the string table is part of the |-delimited array, with a count before the first string, then the payload values in order, rather than reverse), but it is close enough that once you work out the structure of any given type in either request or response, it will be the same for both.

If this is public data, also consider contacting the organization that makes it available, it's always possible they might be able to share more directly?

Finally, depending on the particulars of this, it might be possible to contract with me directly to help write this for you, if that makes sense. If so, please email me at colin@vertispan.com to discuss more.

On Wednesday, October 9, 2024 at 8:50:59 PM UTC-5 cuon...@gmail.com wrote:

Thank to Craig and Colin,

I spent all day to know the stackoverflow post and document Colin provided. Yeah, I know some (not all) rules of request payload, and I can use it to replace param. But the response deserialization is hard, you not mention in you answer, my goal still be parse the response receive from server. 

I need crawl 10000 users via GWT RPC, it is single-time crawl (I crawl it once) for my service. So, the performance is not important.

Again, I know some data from server (response2.png) I attached before, There is a Json, start with //OK, next is Array with 2045 elements (0-2044), element from 0 - 2041 is something to confusing, element 2042 is an array list, it is arranged in a jumble, maybe above data (element from 0-2041) contains order of this.

If you need the exactly response payload, please reply and I public it.

Vào lúc 11:13:42 UTC+7 ngày Thứ Tư, 9 tháng 10, 2024, Craig Mitchell đã viết:

I second what Colin says "Since you're scraping anyway, consider just scraping the results of the rendered page? This will likely take substantially more CPU time, but ridiculously less developer time to implement."

GWT RPC is not an API.  It will constantly change as the website updates.

I'd recommend either using a proper API (if one exists), or an off-the-shelf scraper tool.

On Wednesday 9 October 2024 at 4:40:08 am UTC+11 Colin Alworth wrote:

I'd suggest reading the stream reader/write subtypes of AbstractSerializationStream to understand what all of the values are for - in short, a gwt-rpc response is a payload and a string table, and the payloads elements will reference the string table. You cannot know what the structure is for certain without seeing the original Java types being serialized, but often you can make good guesses.

I'd also suggest reading stackoverflow posts and the like showing how to deserialize other payloads just from context - here's a post that breaks down a payload to understand its contents: https://stackoverflow.com/questions/35047102/serializing-rpc-gwt/35047887#35047887

If you havent yet, read https://docs.google.com/document/d/1eG0YocsYYbNAtivkLtcaiEE5IOF5u4LUol8-LL0TIKU/edit as well.

In short though, your response value is _probably_ be a List of CourseMember types - knowing that class will help you. I can't easily guess more though, as the above doc says, the json array is read backwards, so the important details would be right before and after the string array - you have 1,7,2,1[...strings...] in the second image. From that I can say

1: if this was zero, it would be a null, since it is a positive number, read the (value - 1) entry from the string table, which is ArrayList, so: read a value of type ArrayList from the stream

7: the ArrayList has 7 items

2: first item in the arraylist - as above, if this was 0, it would be null, since it is positive, read the (value - 1) entry from the string table, and decode that type, so: read a CourseMember object from the payload

1: this is _probably_ the number 1 in the first field of the first CourseMember.

...

A parser continuing in this way, with knowledge of the structure of these types could be written to decode this payload. I don't know of an off-the-shelf tool that will do it for you in a truly automated way, but could consult to write one, or guide your project in implementing one by hand.

Since you're scraping anyway, consider just scraping the results of the rendered page? This will likely take substantially more CPU time, but ridiculously less developer time to implement.

On Tuesday, October 8, 2024 at 12:03:30 PM UTC-5 cuon...@gmail.com wrote:

Thank you because the detail response.

I want to crawl data on a public website, I opened devtools and saw that it was written by GWT RPC.

This is the body of request I saw: 

7|0|10|https://a.b.c.d/e|5C6CDB13D0FD25B266F3C36FA7FF6ED9|a1.a2.a3.DataService|getCourseMembers|java.lang.Long/4227064769|java.lang.String/2004016611|java.util.List|20204524|java.util.Arrays$ArrayList/2507071751|20241|1|2|3|4|3|5|6|7|5|TXbrzIAAA|8|9|1|6|10|

As you can see, no problem with that syntax, I can understand roughly, I know the method is getCourseMembers. I want to build a function should return above body, like: 

                  public static String getBodyEncoded(String methodName, ... String methodBody ...) or something similar, and return the body above to send to server.

I also want to know the last past of request syntax:

                 1|2|3|4|1|5|6|7|7|8|7|9|7|10|7|11|7|12|7|13|7|14|

The next is the response body. This is really the problem. A response is very long, I put it in attached files.

I saw a JsonArray with more than 2000 elements, and I cannot understand what are they. The only thing I understand is the 2042nd element, it contains an unorder list. Maybe some elements before contains data about the order.

I want to build a method to extract/deserialize this response.

I am a newbie, if my question can be completed, can you guide me with more details, please?

Java is good, but other languages are acceptable, I still can deploy it.

--
You received this message because you are subscribed to the Google Groups "GWT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit/6f547d38-830d-4e2e-ba90-a16bfe6a781bn%40googlegroups.com.