revlis.nl
Stash of notes about OSS, OSes, virtualization, dev hobby projects &c
October 27, 2023 — 17:16
Xpath or “XML Path Language” is for querying XML and can also be used on HTML DOM. Like selecting ‘div’ elements with a specific class. This can be used when scraping webpages, e.g. with Selenium or Playwright. It works similar to CSS selectors.
Syntax and examples below are all xpath 1.0 since this version is always supported by tools and libs. Version 2.0 adds more types, functions and operators (there’s also 3.0 and 3.1).
Syntax
child::(or'/') selects child (immediate)descendant::selects children (recursive)descendant-or-self::(or'//')@selects attributetext()selects element text
Examples
Select div with ‘myclass’ and ‘title’ attribute
html: <div class="myclass" title="My Title>
xpath: //div[@class="myclass"]/@title
returns: ‘My Title’
Select link with #my_id and then text
html: ‘<a id="my_id">foo bar</a>’
xpath //a[@id="my_id"]/descendant::text()
returns: ‘foo bar’
Testing
Queries can be tested from CLI with ‘xmllint’ (apt install libxml2-utils)
# html file:
xmllint --html --xpath '//a[@class="Results"]/@title' example.html
# actual xml, from curl:
curl http://restapi.adequateshop.com/api/Traveler?page=1 | \
xmllint --xpath '/TravelerinformationResponse/travelers/Travelerinformation/name -
More info
May 28, 2021 — 15:35
If you ever need an IPAM and/or DCIM tool this one is highly recommended. It allows modeling all your infra including network, datacenter and virtualization using a web gui and has an extensive REST API. It can be extended by plugins and "custom fields".
Today it’s widely used and there’s plenty of docs, examples and integrations available.
Installation
The tool is build on Python/Django and uses PostgreSQL. LDAP and other auth methods can be configured. Manual installation includes installing required packages, db and http server. Upgrading to latest version is supported. There’s also Ansible playbooks available for deploying (3rd party).
Objects
There’s Sites, Racks, Devices, Virtualization, VLAN and Interfaces. VM’s and Devices are seen separately and have their own API calls, which might be something to be aware of.
Racks
Devices can be put in racks and have Connections using Cables connecting Interfaces. Same goes for Power, Console, Storage etc.
IPAM
For IPAM there’s Prefixes, IP’s (4 and 6), VLANs, VRF’s and VC’s.
Importing data
Can be done in bulk with e.g. CSV or using the API. If you’re migrating from RackTables there’s ‘racktables2netbox’ but be aware it’s not updated and not directly usable in it’s current state (e.g. API’s changed). It uses ‘pynetbox’, a client lib you can also use for own scripts.
Links
April 3, 2020 — 14:31
Author: silver Category: web Comments: Off
FreshRSS is a self hostable RSS and Atom aggregator (https://freshrss.org) which can also be installed using Docker.
I’m using it myself for Security and general IT related news at https://rss.revlis.nl.
Here’s a Docker run example with persistent data storage and option to edit themes:
docker-freshrss.sh
#!/bin/sh
if ! docker network ls | grep -q freshrss-network; then
docker network create freshrss-network
fi
docker run -d --restart unless-stopped --log-opt max-size=10m \
-v freshrss-data:/var/www/FreshRSS/data \
-v /opt/docker/freshrss/themes:/var/www/FreshRSS/p/themes \
-e 'CRON_MIN=4,34' \
-e TZ=Europe/Amsterdam \
--net freshrss-network \
-p 127.0.0.1:88:80 \
--name freshrss freshrss/freshrss:latest
docker port freshrss
November 25, 2017 — 18:04
GoAccess is a “real-time web log analyzer” which can output in CLI or HTML (like webalizer, awstats and piwik etc). It works out of the box with Apache, for lighttpd you probably need to specify the log format. Examples below are for lighttpd. Run “goaccess /var/log/httpd/access.log” without any other arguments and it will ask for the log format and drop you into the Dashboard (text based gui).
CLI
no conf, just arguments:
goaccess /var/log/lighttpd/access.log \ --date-format=%d/%b/%Y \ --time-format='%T %z' \ --log-format='%h %v %e [%d:%t] "%r" %s %b "%R" "%u"'
-or-
change /etc/goaccess.conf:
date-format %d/%b/%Y:%T %z log-format %h %v %e [%d] "%r" %s %b "%R" "%u"
HTML
Output to “static” html file.
current log:
goaccess /var/log/lighttpd/access.log \ --date-format=%d/%b/%Y \ --time-format='%T %z' \ --log-format='%h %v %e [%d:%t] "%r" %s %b "%R" "%u"' \ --output=/var/www/html/goaccess.html
use all logs:
zcat -f /var/log/lighttpd/access.log*gz | goaccess \ --date-format=%d/%b/%Y \ --time-format='%T %z' \ --log-format='%h %v %e [%d:%t] "%r" %s %b "%R" "%u"' \ --ignore-crawlers \ --with-output-resolver \ -e 127.0.0.1 -e ::1 -e exclude.example.com --output=/var/www/html/goaccess.html
Server
The last option is to run it as Server using WebSocket. This allows it to:
- output realtime HTML:
--real-time-html - run as daemon:
--daemonize - use FIFO:
--fifo-in=--fifo-out= - use HTTPS:
--ssl-cert=--ssl-key= --ws-url=wss://url
live log:
goaccess /var/log/lighttpd/access.log \ --date-format=%d/%b/%Y \ --time-format='%T %z' \ --log-format='%h %v %e [%d:%t] "%r" %s %b "%R" "%u"' \ --output=/var/www/html/goaccess.html \ --real-time-html \ --ssl-cert=//etc/ssl/certs/cert.pem \ --ssl-key=/etc/ssl/private/privkey.pem --ws-url=wss://example.com:7890
Now https://example.com/goaccess.html should should a live Dashboard (tcp port 7890 needs to be open for client).
November 25, 2016 — 12:38
Author: silver Category: web Comments: Off
WP-CLI: A command line interface for WordPress
Download from: https://wp-cli.org/
Configuration:
- /etc/sudoers.d/wp-cli:
- crontab:
March 13, 2012 — 21:31
Author: silver Category: web Comments: Off
To use WordPress on host without ftp/sftp/ssh access, add this line to ‘wp-config.php’
define('FS_METHOD', 'direct');
You must be logged in to post a comment.