database - Retrieving information from aggregated weblogs data, how to do it? -

-

I would like to know how to recover data from the collected logs? This is what I have:
- Uncompressed log data of approximately 30 GB daily daily loaded in HDFS (and it will grow to about 100 GB)
This is my opinion:
- every night it Data is processed with pig

Logs are read, split, and retrieves custom UDF data: timestamp , url , User_id (can say, it is all log-in entry and it loads in Hbiz ( Ogg data will be stored infinitely)

Then if I want to know which user has seen a particular page within a particular time, then I can immediately ask for a complete query with every query Hbiz without scanning the log data (and I want a quick answer - minutes are acceptable) and many more queries are happening at the same time.

What do you think of this workflow? , That this information is HB Base would mean the load? What other options and they'd do I compare my solution? I appreciate all comments / questions and answers. Thank you in advance.

With Hadop you are always doing one of two things (either resources or queries).

I will suggest using HIVE for what you are looking for. You can take your data and then you can prepare an M / R job to process that process from how you like it in the hive table (you can also segment it by data because it is your May be appropriate for motion to see the required data). From here you can ask the results of your data as your choice. Here's a great online tutorial.

There are several ways to solve this, but it seems that HB is a bit taller, unless you want to make it necessary to set up all the servers needed to know it Are there. Hubbies would be nice if you have thousands of people looking to get information together.

You would also like to see in flu which is the new import server from Claudia. This will get your files from some places directly to HDFS

.

Comments

Post a Comment

Popular posts from this blog

c# - sqlDecimal to decimal clr stored procedure Unable to cast object of type 'System.Data.SqlTypes.SqlDecimal' to type 'System.IConvertible' -

-
I'm new to this I'm writing a clr. stored procedure that can be used to get values ​​used in a calculation Call for another stored procedure for Gives an archived methodology int (I think SqlInt32? Value of the table's value is int) which is then necessary to convert to decimal. Here I have the code: Public static integral mood (SQL MDMMYLUE, escalistering AMY type, out of SQLDCM Result) {aResult = 0; Try {SqlConnection conn = New SqlConnection ("Reference Connection = True"); SqlCommand cmd = New SqlCommand ("sp_GetType", conn); Cmd.Parameters.AddWithValue ("@ myValueType", aMyValueType); Cmd.CommandType = CommandType.StoredProcedure; Conn.Open (); Object type = CMD. Exclamation Asser (); Conn.Close (); Decimal type FORCalculation = Convert. ToDecimal (type); Decimal value = convert toDecimal (aMyValue); // ** calculation calc = new calculation (); Decimal result = calorie.docalulation (typefarculation, value); I get an exception...
Read more

Calling GetGUIThreadInfo from Outlook VBA -

-
I need to find the active window'm writing a VBA macro in MS Outlook, which is a specific process. The goal of this is to get the active window / document name even if its original application has more than one document interface. User processor with GetCurrentThreadId is active, I am using GetGUIThreadInfo to get .dll. Here is the correct code: private type RECT as as Left LongPtr top LongPtr right LongPtr down LongPtr termination of private type GUITHREADINFO cbSize as LongPtr flags LongPtr hwndActive LongPtr hwndFocus LongPtr hwndCapture LongPtr as as as as as reset as LongPtr rcCaret as LongPtr hwndCreate as LongPtr hwndMoveSize as HwndMenuOwner end of Private Equity PtrSafe function GetWindowThreadProcessId Lib "user32" (_BVV hnw LongPtr as LongPtr Nig Mr. announced as PtrSafe function GetGUIThreadInfo Lib "user32" _ (ByVal dwthreadid as LongPtr, _ lpguithreadinfo as GUITHREADINFO) LongPtr Sub MyFunction () as dim strWindowTitle string = space as s...
Read more

Obfuscating Python code? -

-
I'm looking for a way to hide my Python source code. print "Hello World!" How can I encode this example so that it is not human-readable? I have been asked to use Base 64 but I'm not sure how. itemprop = "text"> It is limited only, first level ambiguous resolution, but has built-in: byte-code in Python Is a compiler for: dragon-o-m py_compile & lt; Your program.py & gt; creates a .pyo file with a byte-code, and where the docstrings are removed, etc. You can rename .pyo with the .py extension, and python & lt; Your program.py & gt; runs as your program, but it does not have your source code. PS : If your program blur modules like this, you need to rename them with the .pyc suffix (I Not sure it will not break a day), or you can work with .pyo and run them with python -o ... .pyo ( Import should work). This will allow dragon to search your modules (otherwise, Python looks for the .py module).
Read more