Econf World Wide Web

Table of Contents

Introduction

This is some fo the code I use for search engine stuff in Emacs.

Dependencies

Dependency Description
request A popular library for making requests in Emacs Lisp.
w3m A wrapper around the w3m web browser that exits for Emacs.
ace-link A thing that dynamically assigns links key sequences that you can jump to. 
(ec-load-deps deps)

Variables

Here I set up a bunch of variables that are used in the rest of the package. However, I do not set all my variables here and do make some closer to where they are used. 1 I am not sure that this is the best practice, but I am so far from the "best practices" zone with this configuration that I may as well just go with it.

Setting Browsers (and stuff)

(defvar ec-graphical-browser "firefox"
    "The browser and options used to browse the web when a non-gui browser will not work.")

(defvar ec-search-engine "lite.duckduckgo.com/lite/&q="
    "The search engine string used to search the web.")

(defvar ec-video-fstring "vlc '%s'  --qt-minimal-view --no-video-title --loop"
  "The format string used to generate the video playing command.")

(defvar ec-graphical-browse-domains '("http://127.0.0.1:1337?token="
                                      "https://meet.jit.si"
                                      "http://localhost:35901/")
  "Domains that are to be browsed graphically in `w3m-goto-url-new-session-or-video'.")

The w3m Directory

Like everything else I like to keep the data for w3m in my data directory instead of randomly floating around in the dotfiles, so I change it to the data directory here. 

  (defvar ec-w3m-dir (concat ec-data-dir "w3m/")
    "Directory where all w3m files are located.")

Here I set the variables that I have found that need a directory to dump their files in.

Variable Value
w3m-profile-directory ec-w3m-dir
w3m-default-save-directory ec-downloads-dir
w3m-external-view-temp-directory ec-w3m-dir
w3m-bookmark-file (concat ec-w3m-dir "bookmarks")
w3m-cookie-file (concat ec-w3m-dir "cookie") 
(ec-set-variables vars-data-dir)

Further Customizations

Now I just add some further quick customizations to make the browser a little nicer to work with and to conform more with my standard interface.

Variable Value
browse-url-browser-function #'w3m-goto-url-new-session-or-video
w3m-default-display-inline-images t
w3m-pop-up-windows t
w3m-search-default-engine "duckduckgo"
w3m-use-tab nil
w3m-display-mode 'plain 
(ec-set-variables vars-further-custom)

Playing Videos Without JS

Now I don't really like exiting Emacs, so I have added some features to view videos without javascript.

Playing Videos

ec-w3m-view-video

This function simply calls the view video function on the video. 

(defun ec-w3m-view-video (url &rest _)
  (interactive (browse-url-interactive-arg "URL: "))
  (start-process-shell-command "video"
                               nil
                               (format ec-video-fstring url)))

ec-w3m-video-url?

Checks if the video is on a streaming website or if the video content type is something playable with w3m. This way I can directly view videos from download links. 

(defun ec-w3m-video-url? (url)
  "Determine if a URL is a video, if yes return T, if not return nil."
  (and url (or (s-matches? "^\\(http\\|https\\)://www.youtube.com/watch\\?v=.*$" url)
               (s-matches? "^\\(http\\|https\\)://vimeo.com/[0-9]+$" url)
               (member (ec-get-content-type
                        url)
                       '("video/mpg" "video/mpeg" "video/mp3" "video/mp4"
                         "video/avi" "video/wmv"  "video/wav" "video/mov"
                         "video/flv" "video/ogm"  "video/ogg" "video/mkv"
                         "video/webm")))))

Redefining Some Functions

Unfortunately, depending on how a video is reached, a different function may be called. I used to just advise the relevant functions, but I have (against the usual recommendations of the Emacs community) come to prefer redefining functions as redefinition ensures that the function at hand works as expected. 2 Basically advices are weird and difficult to parse when you get too many of them whereas redefinition is pretty clear and simple.

w3m-goto-url-new-session-or-video

This opens a buffer in w3m, a graphical web browser, or vlc intelligently. Exciting. 

(defun w3m-goto-url-new-session-or-video (url &rest _)
  (cond  ((ec-w3m-video-url? url)
          (ec-w3m-view-video url))
         ((->> ec-graphical-browse-domains
            (-map (lambda (domain) (s-contains? domain url)))
            first)
          (ec-graphical-browse-url url))
         (t (w3m-goto-url-new-session url))))

w3m-view-this-url

This just redefines the w3m URL thing entirely, checking if it is a video in the first portion of the cond predicate cond macro. 

(defun w3m-view-this-url (&optional arg new-session)
  "Display the page pointed to by the link under point.
If ARG is the number 2 or the list of the number 16 (you may produce
this by typing `C-u' twice) or NEW-SESSION is non-nil and the link is
an anchor, this function makes a copy of the current buffer in advance.
Otherwise, if ARG is non-nil, it forces to reload the url at point."
  (interactive (if (member current-prefix-arg '(2 (16)))
                   (list nil t)
                 (list current-prefix-arg nil)))
  ;; Store the current position in the history structure.
  (w3m-history-store-position)
  (let ((w3m-prefer-cache
         (or w3m-prefer-cache
             (and (stringp w3m-current-url)
                  (string-match "\\`about://\\(?:db-\\)?history/"
                                w3m-current-url))))
        act url)
    (cond
     ((ec-w3m-video-url? (w3m-anchor))
      (progn (w3m-arrived-intern (w3m-anchor))
             (ec-w3m-view-video (w3m-anchor))
             (w3m-redisplay-this-page)))
     ((setq act (w3m-action))
      (let ((w3m-form-new-session new-session)
            (w3m-form-download nil))
        (ignore w3m-form-new-session w3m-form-download)
        (eval act)))
     ((setq url (w3m-url-valid (w3m-anchor)))
      (if new-session
          (w3m-goto-url-new-session url arg)
        (w3m-goto-url url arg)))
     ((w3m-url-valid (w3m-image))
      (if (display-images-p)
          (w3m-toggle-inline-image)
        (w3m-view-image)))
     ((setq url (w3m-active-region-or-url-at-point 'never))
      (unless (eq 'quit (setq url (w3m-input-url nil url 'quit nil
                                                 'feeling-searchy 'no-initial)))
        (if new-session
            (w3m-goto-url-new-session url arg)
          (w3m-goto-url url arg))))
     (t (w3m-message "No URL at point")))))

Search Engine Suggestions

To add search engine suggestions

Variables

Here I set a list of variables for the search engine suggestions that. 

(defvar *search-history* '("emacs" "clojure" "julia" "logistics")
  "The search history for the current session.")

(defvar *ddg-suggestions* nil
  "The suggestions sent by DDG.")

(defvar *ddg-search-string* nil
  "The string to be searched by DDG.")

Functions

ec-ddg-suggest-fetch

This asynchronously fetches suggestions from the duckduckgo API, using a variable to pass the suggestions to the display function. 

  (defun ec-duckduckgo-suggest-fetch (input)
    (request "https://api.duckduckgo.com/ac/"
      :type "POST"
      :params `(("q" . ,input))
      :parser 'json-read
      :success (cl-function
                (lambda (&key data &allow-other-keys)
                  (setq *ddg-suggestions*
                        (->> data
                             (-map (lambda (val)
                                     (cdar val)))))))))

ec-ddg-get-suggestions

Here we define a pretty simple function to get the list of suggestions for the search engine and merge them with the search history. It also starts a new request for suggestions. 3 Unfortunately this doesn't actually work fully

I also use the rx-macro to dynamically filter out suggestions that don't contain any of the words currently in the input. 

(defun ec-ddg-get-suggestions (input)
  (when (not (s-equals? input *ddg-search-string*))
    (setq *ddg-search-string* input)
    (ec-duckduckgo-suggest-fetch input))
  (->> *search-history*
    (-filter (-partial 's-matches? (rx-to-string `(seq ,@(s-split-words input)))))
    (-union *ddg-suggestions*)))

ec-wikipedia-suggest-fetch

I also like to browse wikipedia. 4 Unfortunately this function doesn't work yet, but soon™, it will. 

(defun ec-wikipedia-suggest-fetch (input)
  (let ((input "emacs"))
    (->> (request "https://en.wikipedia.org/w/api.php"
           :type "GET"
           :params `(("action" . "opensearch")
                     ("format" . "json")
                     ("search" . "emacs"))
           :parser 'json-read
           :sync t)
      request-response-data)))

ec-get-region

This is a simple function to grab a region if it is highlighted, else return nil. IDK if it is defined in emacs elsewhere or not, but I decided to add it anyways because it might not be. 5 Yes, I know that is a terrible idea. 

(defun ec-get-region ()
    "Get the text  or return a empty string if it is not present."
    (if (use-region-p)
        (buffer-substring-no-properties (region-beginning) (region-end))
      ""))

ec-ddg-search-interactive-arg

This searches using my suggestions. I use it mostly to build more complex functions for searching the web. 

  (defun ec-ddg-search-interactive-arg (prompt)
    "Search `w3m-search-default-engine' with ddg completion candidates."
    (let ((ec-ddg-search (ivy-read  prompt
                                     #'ec-ddg-get-suggestions
                                     :initial-input (ec-get-region)
                                     :dynamic-collection t)))
      (setq *search-history* (cons ec-ddg-search *search-history*))
      `(,ec-ddg-search)))

Saving Stuff

I also like to occasionally save documentation, so I have written some functions to make that easier.

ec-clone-website

This just clones a website fully and places it in a directory with the name of the website. 

(defun ec-clone-website (url dirname)
  "Clone a wesbiste to directory."
  (interactive `(,(read-string "URL: ")
                 ,(-> (read-string "Directory name: ")
                    (s-trim))))
  (exec-shell-command (format "httrack \"%s\" -O ~/Websites/%s" url dirname)))

ec-download-video

Sometimes while watching videos from C3 I like to download them, so this is a small, easy function that allows me to do just that. 

  (defun ec-download-video (url &rest _)
    "Download a youtube video."
    (interactive (browse-url-interactive-arg "URL: "))
    (exec-shell-command (concat "youtube-dl -f bestaudio --extract-audio --audio-format mp3 --audio-quality 0 '"
                                url
                                "'")))

User Interface

This is the porcelain of the system. Eventually I think I will separate this into multiple different porcelains (which won't be unit-tested) and libraries (which will).

Buffer Names

Here we add a function to rename the buffers to the current title to make them both more manageable and searchable than they were otherwise. 6 One of my pet-peeves was that w3m didn't really have good names for buffers, instead giving them the usual numeric names which sucked. 

(defun ec-w3m-rename-buffer-name ()
  "Rename the current buffer to whatever the current title is."
  (rename-buffer (format "*world wide web: %s*" w3m-current-title) t))

(add-hook 'w3m-fontify-before-hook #'ec-w3m-rename-buffer-name)

Hydra Interface

We also throw in a hydra interface to make all of this much easier to work with. 

(ec-defhydra hydra-web blue
  ('material "search" "Search the World Wide Web")
  ("Search"
   (("s" ec-search-interactively-new-session"search the web")
    ("u" w3m-goto-url-new-session  "view URL"))
   "Graphical Search"
   (("C-u" ec-graphical-browse-url "view URL using graphical browser")
    ("C-s" ec-graphical-search "search using graphical browser"))
   "Site"
   (("D" (ec-graphical-browse-url "https://discord.com/app") "discord.com/app")
    ("t" (ec-graphical-browse-url "https://twitter.com") "twitter.com")
    ("y" ec-search-yt-interactively "youtube.com"))))

(bind-key (kbd "C-c w") #'hydra-web/body)

Binding Keys

To regularize the user interface we bind a bunch of keys that are, for some ineffable reason, not bound in a standard way. To rebind the keys in the mode I use the table and function call below.

Chord Function
s ec-search-interactively
S ec-search-interactively-new-session
q quit-window
f w3m-view-next-page
b w3m-view-previous-page
n w3m-next-anchor
p w3m-previous-anchor
M-s w3m-select-buffer
<right> forward-char
<left> backward-char
<up> previous-line
<down> next-line
h w3m-history
C-c C-d lexic-search-word-at-point
g w3m-redisplay-this-page
G w3m-reload-this-page
u w3m-goto-url
U w3m-goto-url-new-session 
(ec-bind-keys w3m-bound-keys 'w3m-mode-map)

I also add my system for creating epubs from websites automatically so I can write notes on the websites in org-mode. 

(use-package w3m :config (bind-key (kbd "w")
           (def-web-note-writer w3m-mode w3m-current-url w3m-current-title)
           'w3m-mode-map))

Browser Functions

To make all of it work I also have to define a series of browser functions that will be called in the hydra. 

(defun ec-graphical-browse-url (url &rest _)
  "Browse a URL using the graphical browser."
  (interactive (browse-url-interactive-arg (format "Open URL in %s: " ec-graphical-browser)))  (start-process-shell-command "ec-graphical-browser"
                               nil
                               (format "%s '%s'" ec-graphical-browser url)))

(defun ec-graphical-search (search-terms)
  "Open the graphical browser in a buffer for browsing various websites."
  (interactive (ec-ddg-search-interactive-arg "Search in graphical browser. DuckDuckGo Search: "))
  (exec-shell-command ec-graphical-browser "\"https://" ec-search-engine search-terms "\""))

(defun ec-search-interactively-new-session (search-terms)
  (interactive (ec-ddg-search-interactive-arg "Search in new buffer. DuckDuckGo Search: "))
  (w3m-search-new-session w3m-search-default-engine search-terms))

(defun ec-search-interactively (search-terms)
  (interactive (ec-ddg-search-interactive-arg "Search in current buffer. DuckDuckGo Search: "))
  (w3m-search w3m-search-default-engine search-terms))

(defun ec-search-yt-interactively (search-terms)
  (interactive (ec-ddg-search-interactive-arg "Search in current buffer. Youtube Search: "))
  (w3m-search-new-session w3m-search-default-engine (format "site:youtube.com %s" search-terms)))

Ace Link

Now I just add ace-link to make traversing documents much nicer. 

(ace-link-setup-default)
(bind-key (kbd "C-o") #'ace-link)
(bind-key (kbd "o") #'ace-link 'w3m-mode-map)

Last Modified: 2021-W50-6 00:53

Generated Using: Emacs 27.2 (Org mode 9.4.6)

Except where otherwise noted content on cons.dev is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.