I am stuck trying to automate log parser studio. I've see the jobs that I want to run in the batch manager. Now I am just trying to figure out how to automate things.
I want LPS to load up the IIS logs from the night before, then run the jobs that I've placed in batch manager (I may have to use Task Scheduler), then email me the results.
Is all of this possible?
Using Log Parser Studio, ver 1.01.75 released 3/12/2012.
Users need to log in and validate against the ASP database to get into the website. The user list is small. I have IIS Logfiles that have collected almost all data since the site was online. There is a 3rd party app running on the website, and it produces usage "statistics" that roughly match the number of visits. Using their tool that collects data for only the last 30 days, I was able to approximate the totals using Log Parser Studio.
What I'd like to do is to be able to calculate the approximate length of the users visits on the website, Each user would access the site from only one IP address, so their cs-username & IP would be synonymous. I would like to be able to roll up totals by month and by quarter if possible. This calculation doesn't have to be exact, but I'd like to be able to say that user X was actively on the site for approximately 37 hours last month, as opposed to only 8 1/2 hours for the previous month.
I was using a query like the one below:
/* Manually enter a query or click the library tab */
SELECT Userid
, [cs-username] -- NEWLY ADDED
, [c-ip] -- NEWLY ADDED
, COUNT(*) AS Hits
, FirstHit
, LastHit
USING HASHSEQ(STRCAT(c-ip, [cs(User-Agent)])) AS Userid
, MIN(TO_TIMESTAMP(date, time)) AS FirstHit
, MAX(TO_TIMESTAMP(date, time)) AS LastHit
, EXTRACT_EXTENSION(cs-uri-stem) AS PageType
FROM ex060324.log
WHERE sc-status BETWEEN 200 AND 310
AND PageType NOT IN ('js'; 'css'; 'gif'; 'png')
GROUP BY Userid, [cs-username], [c-ip]
but it only gives me first & last hits, per log file. I've got several hundred logfiles to go through. I think the site has a 20 minute timeout on it. As to "what" constitutes a "visit", each user must hit a login page on a new session (or a timed out one),
so that would be the start. The "end" of the visit would be the last entry on the same day in the logfile prior to the next login.
I haven't been able to find any LogParser Studio queries that would give me the data I'm looking for.
Has anyone developed a query to calculate the visit duration, or to give some approximation of that total? I've also considered importing the raw data in to SQL to see if I could figure out how to do it there. I'd appreciate any input on how to go about accomplishing this rough estimation of usage.
Thanks!
S
I would like to know how the log parser can read and write from a file that looks like the following:
file1_2013-02-14.log. Basically the date changes every day. Basically the date part is year-month-day.
Also can you tell me how to look at yesterday's file and files from 7 days ago?
Hi All,
We are tring to redirect the output of logparser query to SQLite database but we are not getting the expected output .
our command is as follows
logparser.exe -i:FS "select name,path into tblTemp from E:\*.* where attributes not like 'D%'" -o:SQL -database:tempdata.db -driver:"SQLite3 ODBC Driver" -createTable:ON
We get the following output -
Task aborted.
An error occurred while uploading data to SQL table
SQL State: IM001
Native Error: -1
Error Message: not supported
Statistics:
-----------
Elements processed: 140
Elements output: 42
Execution time: 0.01 seconds
the database get created table also get created but no record is inserted into table.
Please help me !!!!
Pravin Thokal.
Hi All,
My apache server log format is as follows..
LogFormat "%{TRUE-CLIENT-IP}i %l %u %t %D \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" \"%{Content-Type}i\" \"%{Cookie}i\"
\"%{Set-Cookie}o\" %{Host}i \"%{Location}o\" " cookie_combined
And below are sample logs...
- - ecomm [10/May/2013:00:15:33 -0500] 60052 "GET /server-status HTTP/1.1" 200 267098 "-" "curl/7.21.0 (x86_64-redhat-linux-gnu) libcurl/7.21.0 NSS/3.12.8.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.4" "-" "-" "-" <IP ADDRESS>
- - - [10/May/2013:00:15:35 -0500] 8787094 "GET /search/results.jsp?Ntt=Sofa+slacks&x=22&y=18 HTTP/1.1" 200 27989 "-" "Mozilla/5.0 (iPad; U; CPU OS 3_2_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B500 Safari/531.21.10""-" "-" "JSESSIONID=v94EgmE82mNuKkDZDBOFvA**.SeaName10; Domain=.mySite.com; Path=/" <IP ADDRESS>
I would like to remove all the request from "ecomm" user and keep remining logs.
I tired following query..
LogParser -i:NCSA "Select * from C:\Logs\www3_access.log WHERE TO_LOWERCASE UserName not like '%ecomm%'"
but getting following error
Task completed with parse errors.
Parse errors:
53335 parse errors occurred during processing (To see details about the parse
error(s), execute the command again with a non-zero value for the "-e"
argument)
Statistics:
-----------
Elements processed: 0
Elements output: 0
Execution time: 0.45 seconds
with -e option
LogParser -i:NCSA "Select * from C:\Logs\www3_access.log WHERE TO_LOWERCASE UserName not like '%ecomm%'" -e:5
Task aborted.
Too many parse errors - aborting
Parse errors:
Log row terminates unexpectedly
LogFile "C:\Logs\www3_access.log", Row number 1
Log row terminates unexpectedly
LogFile "C:\Logs\www3_access.log", Row number 2
Log row terminates unexpectedly
LogFile "C:\Logs\www3_access.log", Row number 3
Log row terminates unexpectedly
LogFile "C:\Logs\www3_access.log", Row number 4
Log row terminates unexpectedly
LogFile "C:\Logs\www3_access.log", Row number 5
Any pointers will be highly appreciated?
Thank You!
PerfGuy!!
Hello, first of all I would like to apologize for my English.
I have a task:
On the Windows XP computer that is running an Internet Information Service (IIS) set up the logging property in such a way that all incoming and outgoing web-traffic gets logged. By the means of Log Parser utility create queries to display the quantity of:
a) sent bytes;
b) received bytes.
I install IIS on XP and set up property for logging web-traffic.
But I realy don't know how create queries...
Plz, help.
Hi all,
I have a syslog server combined with ELSA. I want to parse the RPC alogs along with other logs to the syslog server. My problem is i want this done daily and automaticly.
first i figured i would just make aschduled taks with a bat file where i had my log parser statement in.
So i would parse this path c:\program files\microsoft exchange\logs\rpc\*log
That would parse all the logs content to my syslog server i assume. But my problem is that since im running this job daily wouldn't that mean that i would get double entries since the same log files would be send to the syslog daily?
So i have to make a statement that looks at the date and only parse the specific log file for that day, but how do i do that?
THANKS
Casper
Hello,
i have logs with following structure:
[Tue, 18 Jun 2013 15:55:54 +0200] BE_fcef8956e924520ef278acc1b5885f50a359b8fb xx.xx.xx.xx E_USER_WARNING 0.026 xxx.xx : __soapCall(): LocationDispatcherGetOperatorMembershipSoap.GetOperatorMembership @@12345678|/process_form.php@@ {
WSDL: http://xxx.oxxx.xxx:6180/dispatcher/getOperatorMembershipEndpoint?wsdl
METHOD: GetOperatorMembership
REQUEST:Soap config:
trace:
exceptions: 1
<GetOperatorMembership_Input>
<number>123456789</number>
<header>
<applicationCode>test</applicationCode>
<userId>test</userId>
</header>
and i need parsing fields <number>,<userid> ..Could you somebody help me? Thanks
Hello,
I'd like to know if anyone here needs or uses a utility for watching changes in IIS logs in real-time, with ability to execute custom actions/copy/delete/email notification? We have developed a product for this purpose and we have a limited number of free registration codes available for people willing to give it a test-drive and provide some feedback. If you are interested drop me a note.
BR
So i have tried to export the data from Exchange server to report Outlook client versions connecting and i havent had any luck on that. What i am trying to use is below :
logparser -rtp:-1 "SELECT cs(User-Agent) AS Client, COUNT(*) AS Requests INTO BrowserReport.txt FROM %SystemDrive%\inetpub\logs\LogFiles\W3SVC1\U_ex1108*.log GROUP BY Browser ORDER BY Requests DESC"
and it says Unknow filed "Browser"
I have read this Exchange blog : http://blogs.technet.com/b/exchange/archive/2007/09/12/3403903.aspx and what i am looking for is something simalar to ActiveSync user ID and sort by Device but instead User ID and Outlook Client Version, cs (user-agent) is where the outlook client is logged but i m just finding it hard to export.
Help :(
Hello,
Need your help on Log parser Studio query.
Exchange 2010 SP3
I want to find out number of OWA users connecting to exchange servers hourly. Similarly for Activesync and Outlook anywhere users.
This is required to findout number of concurrent users.
Regards,
Ghouse
hi,
using log parser ; i'm getting output. The output can be put into different file format.
I would like to know is there any way , i can take the output directly into datagridview in c#?
Or
what is the most approproate method to take the log parser output to datagridview in c#?
thank you for the help in advance.
I'm running the following "AvgKBytesPerSec" sample query on a single IIS logfile.
SELECT TO_LOCALTIME(QUANTIZE(time, 1800)) as HalfHour, ADD(SCKBytesSec, CSKBytesSec) as KBytesSec USING DIV(DIV(MUL(1.0, SUM(sc-bytes)), 1024), 1800) as SCKbytesSec, DIV(DIV(MUL(1.0, SUM(cs-bytes)), 1024), 1800) as CSKBytesSec INTO '[OUTFILEPATH]\AvgKBytesPerSec.CSV' FROM '[LOGFILEPATH]' GROUP BY HalfHour ORDER BY HalfHour ASC
I'm getting the following error but have no idea what the problem is.
Specified argument was out of the range of valid values.
Parameter name: i
Can someone shed light on this?
Hi, I just stumbled on logparser and have a very basic beginner question. I have a log file from a FlexLM license server. Can LogParser do anything with this format? Has anyone used LogParser to gather stats on a FlexLM log file?
6:40:19 (mgcld) OUT: "wglibman" 103021994@ES-SGH126QRFV 6:40:19 (mgcld) OUT: "hypbswsiext_c" 312006152@ES-4D6S6BS 6:40:19 (mgcld) OUT: "intexpflow" 312007217@ES-BZPJ6BS 6:40:19 (mgcld) OUT: "wglibman" 103021994@ES-SGH126QRFV 6:40:19 (mgcld) OUT: "hyppower_c" 312006152@ES-4D6S6BS 6:40:19 (mgcld) OUT: "viewdraw" 103021994@ES-SGH126QRFV 6:40:20 (mgcld) OUT: "hyppcbtherm_c" 312006152@ES-4D6S6BS 6:40:20 (mgcld) OUT: "hyplswsiext_c" 312006152@ES-4D6S6BS 6:40:22 (mgcld) OUT: "wglibman" 103021994@ES-SGH126QRFV 6:40:27 (mgcld) OUT: "wgview" 113006932@ES-39FST4J 6:40:31 (mgcld) OUT: "viewdraw" 312007217@ES-BZPJ6BS 6:40:31 (mgcld) OUT: "ices" 312007217@ES-BZPJ6BS 6:40:32 (mgcld) OUT: "dxdatabook" 312007217@ES-BZPJ6BS 6:40:34 (mgcld) OUT: "wgview" r00255346@IP-CGX1W4J 6:40:34 (mgcld) OUT: "wglibman" 103021994@ES-SGH126QRFV 6:40:54 (mgcld) IN: "intexpflow" 312007217@ES-BZPJ6BS 6:41:04 (mgcld) IN: "dxdatabook" 312007217@ES-BZPJ6BS 6:41:05 (mgcld) IN: "ices" 312007217@ES-BZPJ6BS 6:41:05 (mgcld) IN: "viewdraw" 312007217@ES-BZPJ6BS 6:41:07 (mgcld) OUT: "wgview" 113006650@ES-DS3HT4J 6:41:08 (mgcld) OUT: "wgdsncap" r00255346@IP-CGX1W4J 6:41:10 (mgcld) OUT: "wgdsncap" 113006932@ES-39FST4J 6:41:18 (mgcld) OUT: "wglibman" 103021994@ES-SGH126QRFV 6:41:19 (mgcld) OUT: "viewdraw" 312007217@ES-BZPJ6BS 6:41:19 (mgcld) OUT: "ices" 312007217@ES-BZPJ6BS 6:41:21 (mgcld) OUT: "wgdsncap" 113006650@ES-DS3HT4J 6:41:22 (mgcld) UNSUPPORTED: "dxictsheets" (PORT_AT_HOST_PLUS ) 312007217@ES-BZPJ6BS (License server system does not support this feature. (-18,327)) 6:41:24 (mgcld) OUT: "dxdatabook" 312007217@ES-BZPJ6BS 6:41:24 (mgcld) IN: "dxdatabook" 312007217@ES-BZPJ6BS
Hello.. I'm trying to import our Cisco Call logs (CSV file) into SQL. It works, but I'm getting parse errors and it's skipping multiple items; which isn't good.
The Original Called Party Numbers which aren't being imported have '#', '*', and characters in their strings (for e.g. 'b') when LogParser thinks the column is an integer.
To try and resolve this, I've tried using the TO_STRING setting (TO_STRING(originalCalledPartyNumber) as originalCalledPartyNumber), but this still doesn't work. With TO_STRING, get "Error while parsing field originalCalledPartyNumber: Value is not a relative integer".
If I try the REPLACE_STR clause (replace_str(replace_str(originalCalledPartyNumber,'*',''),'#','') as originalCalledPartyNumber), log parser reports "Error: SELECT clause: Semantic Error: 1st argument of function REPLACE_STR must be a STRING".
Any ideas on how I can get these items imported? Thanks..
Pretty new to Logparser, and while I can do some basic things. I'm looking for some examples that might help me with some information I'm trying to query from a log file.
Select Text as LineFromFile FROM Z:\Ingestor_2013-10-15.log WHERE Text Like '%RepoCreator FAILED.%'
Audit 17:02:06 Processing Account: P
Audit 17:02:06 Processing file: F:\Data\STMT_TXT113757.txt.dat
Audit 17:02:06 Setting RepoCreator timeout to 60 minutes
Error 17:02:09 RepoCreator FAILED. Error recieved: 17:02:09 ERROR:
Error 17:02:09 ERROR during processing. Message received:
RepoCreator FAILED. Error recieved: 17:02:09 ERROR:
Error 17:02:09 Function: Processor::InitializeData Line: 748 Number: 0
So I can get the RepoCreator FAILED lines. But I would also like to get the Audit - Processing file line as well to know which file actually failed. The only issue is that if I query for specifically processing file it will also include successfully completed files in the output. I would really like to get the following output.
Audit 17:02:06 Processing file: F:\Data\STMT_TXT113757.txt.dat
Error 17:02:09 RepoCreator FAILED. Error recieved: 17:02:09 ERROR:
Is there a list of index values for say Windows Security EventIDs?
So if I wanted to extract the process name from EventID 4625.
EXTRACT_TOKEN(Strings,18,'|') AS ProcessName
Just wondering if there was a list of values somewhere. That I can use in place of 18 in the code above. Thanks
I am hoping that I just overlooking something basic....
I am attempting to create a ScatterMarkers chart with the following statement:
logparser "SELECT time, time-taken INTO Scatter.png FROM *.log where cs-uri-stem='MyURL.htm" -o:chart -chartType:ScatterMarkers -chartTitle:"Scatter" -i:W3C
But I get the following error message:
"Charts of type 'ScatterMarkers' require series composed of 2 fields each"
If I dump the query into a CSV I get the data I want and can import into Excel and create the Scatter Chart, however, I want to do it all with logparser and cut out the 2nd step with Excel.
Any thoughts are appreciated.
Hi all,
I have a website that has a number of virtual directories in that, running various web services.
I want to use Log Parser to find out how many users are hitting a particular URI per day, and also the the number of unique users per day.
I would appreciate any feedback on how to do this.
Thanks in advance