There is a nice selection of reassuringly non-tacky traditional souvenir shops around Arashiyama station in Western Kyoto, selling fans and traditional sweets. More tacky stores can be found in Gion and the approach to Kiyomizu Temple, selling keyrings, cuddly toys, and garish ornaments. Other traditional souvenirs from Kyoto include parasols and carved wooden dolls.
More unconventional but colorful (and relatively cheap) souvenirs are the wooden votive tablets produced by Shinto shrines, which bear an image relevant to the shrine on the reverse. Visitors write their prayers on the tablets and hang them up, but there's no rule that says you can't take it with you.
Manga and anime enthusiasts should visit Teramachi Street, a covered shopping street off the main Shijo-dori, which boasts a large manga store on two floors, as well as a two-story branch of Gamers (a chain of anime stores), and a small two-story anime and collectables store.
Many ATMs in Kyoto do not allow non-domestic credit cards to be used, but ATMs in post offices and Seven-Eleven usually do. So if you find your card rejected or invalid in an ATM then try and get to a post office (郵便局 / yuubinkyoku or JP (in orange letters)) to use their ATMs instead. Look for the PLUS or Cirrus logos, whichever you find printed on the back of your ATM card. Another option is Citibank, which should work, too. There is an old standby international ATM at the top floor of Takashimaya Department Store at Shijo/Kawaramachi in the "Cash Corner." The bank of ATMs in the basement of the Kyoto Tower shopping center (across the street from JR Kyoto Station) also includes one machine where international cards may be used.