networking - Resuming a failed compression using Java -

-

I have a small number (2-10) large files (6-15 GB) that are really good (4: 1).

I am writing to clients and Java in Java, and I want to send files from the client to the server so that 1. Intermediate Zip file is created)
2. Compressed goods on the server end up in a well-formed file (such as a .zip or .tgz file) so that it can be 'as is' downloadable.
Transfer can be resumed if it fails through the middle path. 4. Restart transfer can be completely in a new process

The first two can be easily obtained by using the java.io socket and Java .util.zip zip output is the third stream that has me Have grief Fourth is actually just the reference.

I'm guessing a solution Maybe some kind of partial re-transmit or a dictionary, or something to be installed will require parse again.

Is there any Java library that supports resumable compression?

I could not find any pre-fab library though the restartable compression in the way I was required There are many bits and pieces available under the open license for writing your own. I have now got a client / server solution that satisfies all the hassles mentioned in the question.

This idea is similar to the chatting ideas given above, but the server manages the chucking and some bookkeeping system compressed section on the client for narrowed chunks on the server. There is no temporary file anywhere in the solution. The basic protocol is as follows: 

 (1) The client sends a manifest to the server, which contains the contents of the zip file (2) the server sends the ID for the server manifest Then again (3) the client asks the server "anything is required for the manifest with id x" (4) server response "no", or with the manifest entry for a file in the manifest, as well Fset and length (5) to send the client compresses the pieces and puts sends him (as well as some bookkeeping information) (6) server each rising zip file, as well as the appropriate zip file crude. If the server orders the volume, then he asks the client properly, all this can be done by file-ended.

The server updates the manifest only, when stage 6 is successfully completed, failure can be safely resumed during steps 3-6 (including crashes on server or client) (Well, more or less).

There are some bits that were a bit shorter to generate a chunky zip file. The basic thing that needs to be acquired is to get a piece-qualified compression algorithm. Deflate can be used in this manner.

The Java ZipOutputstream and the DeflatorAutputstream are not really suitable for 'flywheel' deflation / zipping because they do not allow an arbitrary flushing. ZLib has a BSD-style licensed Java implementation. I did not benchmark it for speed, but it gives the same output as the Java implementation. JZLib is great, and supports all the ZLib flushing modes (unlike java.util.zip.Deflate implementation).

In addition, the zip file calculates the CRC for each entry. Therefore, the manifest entry in step 4 has a 'partial' CRC, which has been updated for each chuck and has been sent back in phased information in step 5. CRC is a public domain implementation for Java, I have benchmarked it and it is fast as the original Java implementation (and provides equivalent CRC).

Finally, the zip file format is quite pernickety. I have been able to work together with an implementation from the Wikipedia page. Although at one point I could not work for the right value for any one area. Fortunately, the ZipOutputTream source is available for JDK so that you can see what they do.


Comments

Post a Comment

Popular posts from this blog

c# - sqlDecimal to decimal clr stored procedure Unable to cast object of type 'System.Data.SqlTypes.SqlDecimal' to type 'System.IConvertible' -

-
I'm new to this I'm writing a clr. stored procedure that can be used to get values ​​used in a calculation Call for another stored procedure for Gives an archived methodology int (I think SqlInt32? Value of the table's value is int) which is then necessary to convert to decimal. Here I have the code: Public static integral mood (SQL MDMMYLUE, escalistering AMY type, out of SQLDCM Result) {aResult = 0; Try {SqlConnection conn = New SqlConnection ("Reference Connection = True"); SqlCommand cmd = New SqlCommand ("sp_GetType", conn); Cmd.Parameters.AddWithValue ("@ myValueType", aMyValueType); Cmd.CommandType = CommandType.StoredProcedure; Conn.Open (); Object type = CMD. Exclamation Asser (); Conn.Close (); Decimal type FORCalculation = Convert. ToDecimal (type); Decimal value = convert toDecimal (aMyValue); // ** calculation calc = new calculation (); Decimal result = calorie.docalulation (typefarculation, value); I get an exception...
Read more

Calling GetGUIThreadInfo from Outlook VBA -

-
I need to find the active window'm writing a VBA macro in MS Outlook, which is a specific process. The goal of this is to get the active window / document name even if its original application has more than one document interface. User processor with GetCurrentThreadId is active, I am using GetGUIThreadInfo to get .dll. Here is the correct code: private type RECT as as Left LongPtr top LongPtr right LongPtr down LongPtr termination of private type GUITHREADINFO cbSize as LongPtr flags LongPtr hwndActive LongPtr hwndFocus LongPtr hwndCapture LongPtr as as as as as reset as LongPtr rcCaret as LongPtr hwndCreate as LongPtr hwndMoveSize as HwndMenuOwner end of Private Equity PtrSafe function GetWindowThreadProcessId Lib "user32" (_BVV hnw LongPtr as LongPtr Nig Mr. announced as PtrSafe function GetGUIThreadInfo Lib "user32" _ (ByVal dwthreadid as LongPtr, _ lpguithreadinfo as GUITHREADINFO) LongPtr Sub MyFunction () as dim strWindowTitle string = space as s...
Read more

Obfuscating Python code? -

-
I'm looking for a way to hide my Python source code. print "Hello World!" How can I encode this example so that it is not human-readable? I have been asked to use Base 64 but I'm not sure how. itemprop = "text"> It is limited only, first level ambiguous resolution, but has built-in: byte-code in Python Is a compiler for: dragon-o-m py_compile & lt; Your program.py & gt; creates a .pyo file with a byte-code, and where the docstrings are removed, etc. You can rename .pyo with the .py extension, and python & lt; Your program.py & gt; runs as your program, but it does not have your source code. PS : If your program blur modules like this, you need to rename them with the .pyc suffix (I Not sure it will not break a day), or you can work with .pyo and run them with python -o ... .pyo ( Import should work). This will allow dragon to search your modules (otherwise, Python looks for the .py module).
Read more