caching an access.log problems

[matthewl]

I am attempting to read the access log for squid and store the last position read so it does not have to reread the file. I am running into many problems with my current code for it. I was wondering if you can help me write a more efficient code and/or help with my current error which is:

Traceback (most recent call last):
  File "./bot.py", line 22, in <module>
    update_cache()
  File "./bot.py", line 14, in update_cache
    mapa = mmap.mmap(f.fileno(),0)
mmap.error: [Errno 13] Permission denied

The permissions for that file is: -rwxrwxrwx 1 nobody nobody 3665356 Dec 17 19:03 /var/logs/squid/access.log

the function for updating the cache of the file:

def update_cache():
  global i
  global mapa
  with open("/var/logs/squid/access.log", 'r') as f:
    f.seek(i)
    mapa = mmap.mmap(f.fileno(),0)
    u = 0
    mapa.write(line)
    u += 1
    print("%d : %s", u, line)
    i = f.tell()
    f.close()

I almost forgot to say I am using python 3.1.1

[richieking]

Can you post your code again with your import modules if any.
Just in short what you have provided is not enough to help assist you.
:)

[matthewl]

here you go. Thanks.

#!/usr/local/bin/python3
import socket, ssl, mmap

bindsocket = socket.socket()
bindsocket.bind(('secure.dwdcweb.info', 8180))
bindsocket.listen(200)
i =0;
mapa = "";
def update_cache():
  global i
  global mapa
  f = open("/var/logs/squid/access.log", 'r+b')
  
  mapa = mmap.mmap(f.fileno(),0)  
  u = 0
  for line in f.read():
    f.seek(i)  
    mapa.write(line)
    u += 1
    print("%d : %s", u, line)
    i = f.tell()
  f.close()

update_cache()
def do_something(connstream, data):
  print("%s",data)
  connstream.write("HTTP/1.1 200 OK\r\n");
  connstream.write("Server: 	Secure Proxy RPC Server/0.1 (CentOS)\r\n");
  connstream.write("Cache-Control: no-cache\r\n");
  connstream.write("Pragma: no-cache\r\n");
  connstream.write("Expires: Thu, 01 Dec 1994 16:00:00 GMT\r\n");
  connstream.write("Content-Type: text/xml\r\n");
  connstream.write("Access-Control-Allow-Origin: http://secure.dwdcweb.info\r\nAccess-Control-Allow-Methods: POST, OPTIONS\r\n");
  connstream.write("Access-Control-Max-Age: 1728000\r\n");
  connstream.write("\r\n\r\n<?xml version=\"1.0\" standalone=\"yes\" ?><row>\r\n");
  connstream.write("</rows>\r\n");

def deal_with_client(connstream):
   data = connstream.read()
   # null data means the client is finished with us
   while data:
      if not do_something(connstream, data):
         
         break
      data = connstream.read()
   # finished with client
   connstream.close()


while True:
   newsocket, fromaddr = bindsocket.accept()
   update_cache()
   connstream = ssl.wrap_socket(newsocket,
                                server_side=True,
                                certfile="/home/dwdataconcepts/domains/secure.dwdcweb.info/ssl.cert",
                                keyfile="/home/dwdataconcepts/domains/secure.dwdcweb.info/ssl.key",
                                ssl_version=ssl.PROTOCOL_SSLv3)
   deal_with_client(connstream)

[richieking]

I come out with this.

We need to trick the system to accept our terms and conditions.
Here You ghost the mmap.mmap with our logical numbers with maintaining the exact anmout of data so that there will not be data trancation.

try this

#!/usr/local/bin/python3
import socket, ssl, mmap

bindsocket = socket.socket()
bindsocket.bind(('secure.dwdcweb.info', 8180))
bindsocket.listen(200)
i =0;
mapa = "";
def update_cache():
  global i
  global mapa
  f = open("/var/logs/squid/access.log", 'r+b')
  f=len(f.read()) # To get the size of the file 
  mapa = mmap.mmap(-1,f) # To trick the system to accept operation  
  u = 0
  for line in f.read():
    f.seek(i)  
    mapa.write(line)
    u += 1
    print("%d : %s", u, line)
    i = f.tell()
  f.close()

update_cache()
def do_something(connstream, data):
  print("%s",data)
  connstream.write("HTTP/1.1 200 OK\r\n");
  connstream.write("Server:     Secure Proxy RPC Server/0.1 (CentOS)\r\n");
  connstream.write("Cache-Control: no-cache\r\n");
  connstream.write("Pragma: no-cache\r\n");
  connstream.write("Expires: Thu, 01 Dec 1994 16:00:00 GMT\r\n");
  connstream.write("Content-Type: text/xml\r\n");
  connstream.write("Access-Control-Allow-Origin: http://secure.dwdcweb.info\r\nAccess-Control-Allow-Methods: POST, OPTIONS\r\n");
  connstream.write("Access-Control-Max-Age: 1728000\r\n");
  connstream.write("\r\n\r\n<?xml version=\"1.0\" standalone=\"yes\" ?><row>\r\n");
  connstream.write("</rows>\r\n");

def deal_with_client(connstream):
   data = connstream.read()
   # null data means the client is finished with us
   while data:
      if not do_something(connstream, data):
         
         break
      data = connstream.read()
   # finished with client
   connstream.close()


while True:
   newsocket, fromaddr = bindsocket.accept()
   update_cache()
   connstream = ssl.wrap_socket(newsocket,
                                server_side=True,
                                certfile="/home/dwdataconcepts/domains/secure.dwdcweb.info/ssl.cert",
                                keyfile="/home/dwdataconcepts/domains/secure.dwdcweb.info/ssl.key",
                                ssl_version=ssl.PROTOCOL_SSLv3)
   deal_with_client(connstream)

If you like it.... lil upvoting will do.
Happy holidays ;)

[richieking]

Updated

import mmap,os

f= open("/home/richie/be.txt","r") 
   
ff=len(f.read())

mm=mmap.mmap(-1,ff)
f.close()
fc=open("/home/richie/be.txt","r") 
mf=fc.read()    
    
mm.write(mf)

mm.seek(0)
   
print mm.read(20000)        
mm.close()


#!/usr/local/bin/python3
import socket, ssl, mmap

bindsocket = socket.socket()
bindsocket.bind(('secure.dwdcweb.info', 8180))
bindsocket.listen(200)
i =0;
mapa = "";
def update_cache():
  global i
  global mapa
  f = open("/var/logs/squid/access.log", 'r+b')
  f=len(f.read()) # To get the size of the file 
  mapa = mmap.mmap(-1,f) # To trick the system to accept operation  
  f.close() # For strange reasons The file needs to closw and reopen. I just check that out
            # else it produces nothing for mmap writting
  u = 0
  f1=open("/var/logs/squid/access.log", 'r+b')
  for line in f1.read():      # You can also read the entire line into the mmap
    f1.seek(i)                # mapa.write(f1.read()) So there is no loop there
    mapa.write(line)          # Because the loop will not get you the actual lines in the file
    u += 1                    # If you want the actual line 100% This is not the solution
    print("%d : %s", u, line) #I think you need to use realines() loop on the mapa after as 
    i = f1.tell()             # This mapa write strip and make the file more clean and compact.
  f1.close()

update_cache()
def do_something(connstream, data):
  print("%s",data)
  connstream.write("HTTP/1.1 200 OK\r\n");
  connstream.write("Server:     Secure Proxy RPC Server/0.1 (CentOS)\r\n");
  connstream.write("Cache-Control: no-cache\r\n");
  connstream.write("Pragma: no-cache\r\n");
  connstream.write("Expires: Thu, 01 Dec 1994 16:00:00 GMT\r\n");
  connstream.write("Content-Type: text/xml\r\n");
  connstream.write("Access-Control-Allow-Origin: http://secure.dwdcweb.info\r\nAccess-Control-Allow-Methods: POST, OPTIONS\r\n");
  connstream.write("Access-Control-Max-Age: 1728000\r\n");
  connstream.write("\r\n\r\n<?xml version=\"1.0\" standalone=\"yes\" ?><row>\r\n");
  connstream.write("</rows>\r\n");

def deal_with_client(connstream):
   data = connstream.read()
   # null data means the client is finished with us
   while data:
      if not do_something(connstream, data):
         
         break
      data = connstream.read()
   # finished with client
   connstream.close()


while True:
   newsocket, fromaddr = bindsocket.accept()
   update_cache()
   connstream = ssl.wrap_socket(newsocket,
                                server_side=True,
                                certfile="/home/dwdataconcepts/domains/secure.dwdcweb.info/ssl.cert",
                                keyfile="/home/dwdataconcepts/domains/secure.dwdcweb.info/ssl.key",
                                ssl_version=ssl.PROTOCOL_SSLv3)
   deal_with_client(connstream)

[matthewl]

thanks.

the first post had these errors:

Traceback (most recent call last):
  File "./bot.py", line 24, in <module>
    update_cache()
  File "./bot.py", line 16, in update_cache
    for line in f.read():
AttributeError: 'int' object has no attribute 'read'
-sh-3.2$ ./bot.py
Traceback (most recent call last):
  File "./bot.py", line 24, in <module>
    update_cache()
  File "./bot.py", line 16, in update_cache
    for line in f.read():
AttributeError: 'int' object has no attribute 'read'

these are some of the errors your last post gave me. I removed the test code above the #!/usr/local/bin/python3. So that way it is line starts at line seven.

File "./bot.py", line 15, in update_cache
    f.close() # For strange reasons The file needs to closw and reopen. I just check that out
AttributeError: 'int' object has no attribute 'close'
-sh-3.2$ ./bot.py
import: unable to open X server `'.
./bot.py: line 7: syntax error near unexpected token `('
./bot.py: line 7: `bindsocket = socket.socket()'
-sh-3.2$ ./bot.py
import: unable to open X server `'.
./bot.py: line 7: syntax error near unexpected token `('
./bot.py: line 7: `bindsocket = socket.socket()'
-sh-3.2$ ./bot.py
Traceback (most recent call last):
  File "./bot.py", line 27, in <module>
    update_cache()
  File "./bot.py", line 15, in update_cache
    f.close() # For strange reasons The file needs to closw and reopen. I just check that out
AttributeError: 'int' object has no attribute 'close'

[richieking]

Add the imports

import socket, ssl, mmap

This code is working fine with me.

Add the imports on the top of my updated post.
The only problem was the mmap which the ghost hack works fine. I have use a prototype here and its all good.

;)