revlis.nl

October 27, 2023 — 17:16

Author: silver Category: dev web Comments: Off

Xpath or “XML Path Language” is for querying XML and can also be used on HTML DOM. Like selecting ‘div’ elements with a specific class. This can be used when scraping webpages, e.g. with Selenium or Playwright. It works similar to CSS selectors.

Syntax and examples below are all xpath 1.0 since this version is always supported by tools and libs. Version 2.0 adds more types, functions and operators (there’s also 3.0 and 3.1).

Syntax

child:: (or '/') selects child (immediate)
descendant:: selects children (recursive)
descendant-or-self:: (or '//')
@ selects attribute
text() selects element text

Examples

Select div with ‘myclass’ and ‘title’ attribute

html: <div class="myclass" title="My Title>

xpath: //div[@class="myclass"]/@title

returns: ‘My Title’

Select link with #my_id and then text

html: ‘<a id="my_id">foo bar</a>’

xpath //a[@id="my_id"]/descendant::text()

returns: ‘foo bar’

Testing

Queries can be tested from CLI with ‘xmllint’ (apt install libxml2-utils)

# html file:
xmllint --html --xpath '//a[@class="Results"]/@title' example.html
# actual xml, from curl:
curl http://restapi.adequateshop.com/api/Traveler?page=1 | \
  xmllint --xpath '/TravelerinformationResponse/travelers/Travelerinformation/name -

More info

NetBox

May 28, 2021 — 15:35

Author: silver Category: network web Comments: Off

If you ever need an IPAM and/or DCIM tool this one is highly recommended. It allows modeling all your infra including network, datacenter and virtualization using a web gui and has an extensive REST API. It can be extended by plugins and "custom fields".

Today it’s widely used and there’s plenty of docs, examples and integrations available.

Installation

The tool is build on Python/Django and uses PostgreSQL. LDAP and other auth methods can be configured. Manual installation includes installing required packages, db and http server. Upgrading to latest version is supported. There’s also Ansible playbooks available for deploying (3rd party).

Objects

There’s Sites, Racks, Devices, Virtualization, VLAN and Interfaces. VM’s and Devices are seen separately and have their own API calls, which might be something to be aware of.

Racks

Devices can be put in racks and have Connections using Cables connecting Interfaces. Same goes for Power, Console, Storage etc.

IPAM

For IPAM there’s Prefixes, IP’s (4 and 6), VLANs, VRF’s and VC’s.

Importing data

Can be done in bulk with e.g. CSV or using the API. If you’re migrating from RackTables there’s ‘racktables2netbox’ but be aware it’s not updated and not directly usable in it’s current state (e.g. API’s changed). It uses ‘pynetbox’, a client lib you can also use for own scripts.

Links

FreshRSS

April 3, 2020 — 14:31

Author: silver Category: web Comments: Off

FreshRSS is a self hostable RSS and Atom aggregator (https://freshrss.org) which can also be installed using Docker.

I’m using it myself for Security and general IT related news at https://rss.revlis.nl.

Here’s a Docker run example with persistent data storage and option to edit themes:

docker-freshrss.sh

#!/bin/sh
if ! docker network ls | grep -q freshrss-network; then
  docker network create freshrss-network
fi
docker run -d --restart unless-stopped --log-opt max-size=10m \
  -v freshrss-data:/var/www/FreshRSS/data \
  -v /opt/docker/freshrss/themes:/var/www/FreshRSS/p/themes \
  -e 'CRON_MIN=4,34' \
  -e TZ=Europe/Amsterdam \
  --net freshrss-network \
  -p 127.0.0.1:88:80 \
  --name freshrss freshrss/freshrss:latest
docker port freshrss

GoAccess

November 25, 2017 — 18:04

Author: silver Category: linux web Comments: Off

GoAccess is a “real-time web log analyzer” which can output in CLI or HTML (like webalizer, awstats and piwik etc). It works out of the box with Apache, for lighttpd you probably need to specify the log format. Examples below are for lighttpd. Run “goaccess /var/log/httpd/access.log” without any other arguments and it will ask for the log format and drop you into the Dashboard (text based gui).

CLI

no conf, just arguments:

goaccess /var/log/lighttpd/access.log \
--date-format=%d/%b/%Y \
--time-format='%T %z' \
--log-format='%h %v %e [%d:%t] "%r" %s %b "%R" "%u"'

-or-

change /etc/goaccess.conf:

date-format %d/%b/%Y:%T %z
log-format %h %v %e [%d] "%r" %s %b "%R" "%u"

HTML

Output to “static” html file.

current log:

goaccess /var/log/lighttpd/access.log \
  --date-format=%d/%b/%Y \
  --time-format='%T %z' \
  --log-format='%h %v %e [%d:%t] "%r" %s %b "%R" "%u"' \
  --output=/var/www/html/goaccess.html

use all logs:

zcat -f /var/log/lighttpd/access.log*gz | goaccess \
  --date-format=%d/%b/%Y \
  --time-format='%T %z' \
  --log-format='%h %v %e [%d:%t] "%r" %s %b "%R" "%u"' \
  --ignore-crawlers \
  --with-output-resolver \
  -e 127.0.0.1 -e ::1 -e exclude.example.com
  --output=/var/www/html/goaccess.html

Server

The last option is to run it as Server using WebSocket. This allows it to:

output realtime HTML: --real-time-html
run as daemon: --daemonize
use FIFO: --fifo-in= --fifo-out=
use HTTPS: --ssl-cert= --ssl-key= --ws-url=wss://url

live log:

goaccess /var/log/lighttpd/access.log \
 --date-format=%d/%b/%Y \
 --time-format='%T %z' \
 --log-format='%h %v %e [%d:%t] "%r" %s %b "%R" "%u"' \
 --output=/var/www/html/goaccess.html \
 --real-time-html \
 --ssl-cert=//etc/ssl/certs/cert.pem \
 --ssl-key=/etc/ssl/private/privkey.pem --ws-url=wss://example.com:7890

Now https://example.com/goaccess.html should should a live Dashboard (tcp port 7890 needs to be open for client).

WP-CLI

November 25, 2016 — 12:38

Author: silver Category: web Comments: Off

WP-CLI: A command line interface for WordPress

Download from: https://wp-cli.org/

Configuration:

/etc/sudoers.d/wp-cli:

# wordpress update:
username ALL=(ALL:ALL) NOPASSWD: /usr/bin/php5 /usr/local/bin/wp --allow-root --path=/var/www/site.com cli update --yes
username ALL=(ALL:ALL) NOPASSWD: /usr/bin/php5 /usr/local/bin/wp --allow-root --path=/var/www/site.com core update
username ALL=(ALL:ALL) NOPASSWD: /usr/bin/php5 /usr/local/bin/wp --allow-root --path=/var/www/site.com plugin update --all
username ALL=(ALL:ALL) NOPASSWD: /usr/bin/php5 /usr/local/bin/wp --allow-root --path=/var/www/site.com theme update --all

crontab:

0 2 * * * echo "$( date +\%F\ \%\T ) wp: $( sudo /usr/bin/php5 /usr/local/bin/wp --allow-root --path=/var/www/site.com cli update --yes )" >> /var/log/wp-cli 2>&1
5 2 * * * echo "$( date +\%F\ \%\T ) wp: $( sudo /usr/bin/php5 /usr/local/bin/wp --allow-root --path=/var/www/site.com core update )" >> /var/log/wp-cli  2>&1
15 2 * * * echo "$( date +\%F\ \%\T ) wp: $( sudo /usr/bin/php5 /usr/local/bin/wp --allow-root --path=/var/www/site.com plugin update --all )" >> /var/log/wp-cli 2>&1
20 2 * * * echo "$( date +\%F\ \%\T ) wp: $( sudo /usr/bin/php5 /usr/local/bin/wp --allow-root --path=/var/www/site.com theme update --all )" >> /var/log/wp-cli 2>&1

WP without (s)ftp/ssh

March 13, 2012 — 21:31

Author: silver Category: web Comments: Off

To use WordPress on host without ftp/sftp/ssh access, add this line to ‘wp-config.php’

define('FS_METHOD', 'direct');